Repository navigation
SYCL batching causes invalid results #445
Description
Hello, I am trying to run a simplified "forward" passage of a neural network with GPU.
On CUDA/CuArray I have always the same, correct results for my output, but with oneAPI/oneArray, the first time I have the correct result, but the subsequent times I have random results and random crashes (but never the first time).
Any clue ?
using Test, oneAPI, BenchmarkTools, LinearAlgebra
# Function definitions
relu(x) = max(0,x)
forward_layer(x,w,w0,f) = f.(w*x .+ w0)
function forward_network!(y,x,w1,w2,w3,w01,w02,w03,f=relu)
x1 = forward_layer(x,w1,w01,f)
x2 = forward_layer(x1,w2,w02,f)
y .= forward_layer(x2,w3,w03,identity)
return nothing
end
# CPU data
(nd0,nd1,nd2,ndy) = (200,300,300,1)
x = rand(Float32,nd0); y = Vector{Float32}(undef,ndy)
w1 = rand(Float32,nd1,nd0); w2 = rand(Float32,nd2,nd1); w3 = rand(Float32,ndy,nd2)
w01 = rand(Float32,nd1); w02 = rand(Float32,nd2); w03 = rand(Float32,ndy);
# CPU call
forward_network!(y,x,w1,w2,w3,w01,w02,w03,relu)
# GPU data
y_g = oneArray{Float32}(undef,ndy)
x_g = oneArray(x)
w1_g = oneArray(w1); w2_g = oneArray(w2); w3_g = oneArray(w3);
w01_g = oneArray(w01); w02_g = oneArray(w02); w03_g = oneArray(w03);
# GPU call
forward_network!(y_g,x_g,w1_g,w2_g,w3_g,w01_g,w02_g,w03_g,relu)
# Correctness check..
y ≈ Array(y_g) # true
# Second (and further) attempt..
y_g = oneArray{Float32}(undef,ndy)
forward_network!(y_g,x_g,w1_g,w2_g,w3_g,w01_g,w02_g,w03_g,relu)
y ≈ Array(y_g) # false !Perhaps linked to #327 ?
Ubuntu 22.04, oneAPI v1.5.0, Intel CPU i5-8350U, UHD Graphics 620
Activity
maleadt commentedon Jul 5, 2024
MWE for the correctness issue:
Crashes I've seen:
And during process exit:
maleadt commentedon Jul 5, 2024
Looks like setting
SYCL_PI_LEVEL_ZERO_BATCH_SIZEfrom__init__doesn't work, as we've already loaded MKL then and SYCL doesn't re-parse the environment variable (which makes sense). As a result, there's outstanding SYCL operations that aren't materialized by oursynchronize().@pengtu Is there perhaps a different way to change the batch size? It would require a bit of engineering across the Julia BinaryBuilder stack to make it possible to set env vars when the library is
dlopened.[-]Random result and crash in running a function twice (oneArray)[/-][+]SYCL batching causes invalid results[/+]maleadt commentedon Oct 19, 2026
Fixed by #563, which makes every oneMKL wrapper wait on its SYCL queue so batched SYCL work is always submitted, together with the stream/oneMKL ordering from #610. The MWE and the original network now give correct results on every iteration.