Repository navigation
POCL: cheaper host kernel launches - #941
Merged
Merged
Conversation
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
October 28, 2026 09:31
138e60c to
3aa7c96
Compare
Contributor
Benchmark resultsCompared with 8a985320e, the latest measured ancestor of the merge base (7 commits before it). Times are the minimum time of every benchmark. A change is unusual when it exceeds both three times the benchmark's typical change between consecutive runs on the default branch and a minimum per category (performance 3%, latency 5%, applications 3%, other 3%). A benchmark that starts or stops allocating is listed however few the allocations. A single run can still be a fluke, so rerun before drawing conclusions.
All benchmarks
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## tb/pocl-host-kernel-bench #941 +/- ##
=============================================================
+ Coverage 84.62% 84.81% +0.18%
=============================================================
Files 31 31
Lines 2901 2924 +23
=============================================================
+ Hits 2455 2480 +25
+ Misses 446 444 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vchuravy
reviewed
Oct 30, 2026
vchuravy
reviewed
Oct 30, 2026
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 4, 2026 09:12
3aa7c96 to
1f10ede
Compare
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 5, 2026 08:31
1f10ede to
c5f18fa
Compare
maleadt
added a commit
to vchuravy/ThreadTeams.jl
that referenced
this pull request
Nov 5, 2026
…#87) While it waits for the participants, the caller of a range region yields: in the park branch of `rescue_ranges`, every 1024 spins of `wait_idle`, and in `notify` when another task holds the event's lock. Tasks queued on its thread then run in the middle of the call. With `isolate = true` the caller only spins and parks its OS thread, and wakes workers after taking the event's lock with `trylock`, so it switches tasks only where `f` does. Should a call of `f` that yielded move the caller onto another worker's thread, the waits yield again, since that worker could otherwise not progress. The other loops and the default are unchanged. A caller needs this when it relies on state of its thread across the call. On Julia up to 1.13, `disable_sigint` defers a Ctrl-C only while the deferring task runs on thread 1: when it yields, the interrupt is thrown into whichever task runs there next, usually the thread-1 worker, which dies (if it had not started yet) or drops it (#40). KernelAbstractions' CPU back-end defers interrupts for the whole of a kernel launch on the team, and lost them this way (JuliaGPU/KernelAbstractions.jl#941).
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 5, 2026 16:15
c5f18fa to
be71e2b
Compare
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 6, 2026 08:00
be71e2b to
a547dcd
Compare
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 7, 2026 11:42
a547dcd to
6d2f350
Compare
Avoid repeated struct copies and store-forwarding stalls. A trivial launch on a Xeon E3-1245 v6 drops from 602 to 566 cycles.
Use shifts and masks for power-of-two work-group sizes; skip division by the extent in preliminary partitions. This reduces repeated divisions in partitioning, coverage checks and PoCL tuning. On a Xeon E3-1245 v6, where 64-bit division costs tens of cycles, a trivial launch drops from 566 to 493 cycles.
Reuse the predicted size and coverage result when the backend confirms the prediction. PoCL checks that kernel and device limits match. Avoiding repeated tuning and coverage checks saves about 85 instructions per launch.
Use the vector-based helper shared with `launch_converted` to avoid recompiling the `host_launch_converted` generator for each argument count. Its expression tuples cost 15-20 ms per compilation, 59 times in an Oceananigans model's first time step.
Cycle-counter reads cost 40 of a trivial launch's 383 cycles on a Ryzen 9 5950X; on a Xeon E3-1245 v6, they overlap with the launch. After a sample below 1 µs, time one in 16 launches. Always time grids with no estimate or a changed work-group count. Detecting longer work within a grid can lag by 16 launches; work varying with that period can evade sampling entirely.
Share argument conversion between queue and host-kernel paths to avoid duplicating most per-kernel launch code. Launch inference for the 71 kernels of an Oceananigans time step drops from 5.6 to 3.1 s.
Call PoCL directly for unsplit launches. This avoids the memory traffic from passing `HostLaunch` by reference to `host_run`.
Member
Author
|
CI failure is JuliaLang/julia#62471, so unrelated. |
maleadt
added a commit
that referenced
this pull request
Nov 7, 2026
On macOS, Julia before 1.13 installs SIGINT's handler on the main thread while the signal listener thread sets it to SIG_IGN for kqueue. When the listener loses that race, the process ignores every SIGINT, and all six interrupt testsets fail with `err === nothing` (seen on #941's macOS 1.10 CI). Fixed upstream in JuliaLang/julia#62471, which only 1.13 has.
maleadt
force-pushed
the
tb/pocl-launch-cost-906
branch
from
November 7, 2026 14:46
6d2f350 to
cc7cf19
Compare
maleadt
added a commit
that referenced
this pull request
Nov 8, 2026
* POCL: run kernels on Julia's threads with PoCL host kernels Handing a launch to PoCL's thread pool and waiting for it costs about 5 µs, even for an empty kernel. PoCL's `cl_pocl_host_kernel` extension, in every pocl_standalone_jll since 7.2.3, runs work-groups on the calling thread instead. On a 4-core x86 machine, a launch now costs about 0.15 µs on the launching task and 1 µs on a ThreadTeams team. Launches of up to 8192 work-items stay on the launching task unless the kernel's recent launches were slow. Larger ones use `foreach_range`, which starts block `k` of a `Threads.@threads :static` split on thread `k`, keeping data in place across launches; threads steal work when the grid changed or the previous launch took 50 µs or more. ThreadTeams 1.3 steals from nearby threads first. Host kernels can't yield: interrupts are deferred until the launch ends, and tasks, timers and I/O run after team and long launches. Runs are GC-safe, and launches from finalizers, which can't switch tasks, run every work-group on the calling thread. The command queue remains for explicit work-group sizes over the 1 MiB stack limit, kernels PoCL can't prepare, and direct POCL calls with non-uniform sizes. The thread count is capped at Julia's default pool. * POCL: run only partial work-groups with the bounds check Only partial work-groups need the bounds check, but a launch with any ran the checked kernel, whose work-item loop vectorizes poorly, for all of them. Run the full ones with a variant compiled without the check, in the same launch, sharing arguments, geometry and exception status. Linear launches, launches on the command queue, and launches past three dimensions with a partial third dimension stay checked throughout. A kernel's first launch with partial work-groups stays checked, so that kernels used once, such as AcceleratedKernels closures, compile once: on 4 Xeon E3-1245 v6 threads, a trivial kernel's first launch takes 122 ms, as through PoCL's queue, instead of 167 ms. If the unchecked variant can't run as a host kernel, the checked one runs every work-group. * POCL: fit tuned work-groups to host kernel launches Work-groups whose stack estimate exceeds 1 MiB run on PoCL's command queue instead of as host kernels. WaterLily's donut body kernel (344 B per work-item, 4096 work-items on Zen 5) did so every step. Cap tuned work-groups, once per kernel at link time, at the largest power of two that fits; explicit sizes still fall back. On 16 Zen 5 threads, donut_5 takes 2.2-2.35 ms per step instead of 2.9-3.0, and donut_6 11.3-13.2 ms instead of 14.4-15.0. Fitting work-groups to the ndrange also left small work-item loops with scalar remainders: axpy over 1000 work-items on 32 threads took 162 ns with groups of 25 instead of 146 ns with groups of 32. Shrink a first dimension below 16 vectors only to a vector-width multiple. * Document how the CPU backend runs kernels on Julia's threads Describe which thread runs which work-groups, when a launch stays on the launching task or lets other tasks run, and which kernels still run on PoCL's threads. Pin threads with ThreadPinning.jl rather than `POCL_AFFINITY`; the NUMA example and its measurements now do so. * POCL: keep deferred interrupts with team launches A launch defers `^C` until all work-groups finish, but ThreadTeams' dispatcher yields while it waits for the team. On Julia 1.13 and earlier, a task switch delivers the deferred SIGINT to whichever task runs next on thread 1, usually ThreadTeams' worker there, so the launch returned normally and the interrupt was lost. Dispatch with ThreadTeams 1.4's `isolate = true`, which never switches tasks, on all versions. The first scratch slot lookup on a thread may also take a lock and switch tasks; create the slots on the launching task before a team launch starts. Document that before Julia 1.14 only launches from thread 1 are protected, and test a launch with a task waiting on its thread. * Test: allow more team runs before a cheap kernel turns serial The small host launches test expected one of five team launches of a trivial kernel to estimate a serial duration under 4 µs. A team run times its first range on a team thread, which on CI may have just woken up or been descheduled: the macOS runners have 3 cores for 5 Julia threads. One CI run measured five slow ranges in a row, so the kernel stayed on the team. Launch until the schedule turns serial, up to 100 times; the first fast run is enough. * Test: skip interrupted launches when the process ignores SIGINT On macOS, Julia before 1.13 installs SIGINT's handler on the main thread while the signal listener thread sets it to SIG_IGN for kqueue. When the listener loses that race, the process ignores every SIGINT, and all six interrupt testsets fail with `err === nothing` (seen on #941's macOS 1.10 CI). Fixed upstream in JuliaLang/julia#62471, which only 1.13 has.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The KernelInterface size-check optimization is in #938, which bumps KernelInterface to 0.4.4.
Reduce PoCL host-kernel launch overhead: return the session by reference, avoid 64-bit divisions in launch geometry, tune work-group sizes once, and sample short-launch timings. Unsplit launches on the calling thread call PoCL directly, avoiding launch materialization. Sharing argument generation and conversion between queue and host-kernel paths reduces first-launch compilation.
Inline launches (
icount.jl): median ns over 6 runs, #906 → this PR with #938 (Julia 1.12.7, ThreadTeams 1.3.0):-t 1-t 4,0-t 1-t 16,0Trivial launch on the Xeon: 1706 → 1401 instructions, 633 → 459 cycles.
Without #938, a launch on the Xeon is ~31 instructions, ~24 cycles (4–6 ns) slower (e.g. trivial launch: 1432 instructions, 486 cycles, 119.4 ns).
First launches in a fresh process (Ryzen 9950X,
-t 4, median of 3, ms): inline 667 → 573, stencil 795 → 617, axpy 237 → 161. Subsequent new kernels are unchanged (65–70 ms each).Trade-off: after a sample below 1 µs, time one in 16 launches. Detecting longer work within a grid can lag by 16 launches; work varying with that period can evade sampling entirely. Always time grids with no estimate or a changed work-group count.