[maleadt]

Tracking PR for cl_pocl_host_kernel, a draft extension that lets an application run a kernel's work-groups on its own threads: synchronously, without command queues, events, locks or allocation. Not proposed upstream yet; it exists to track the work and to let people benchmark it.

Why. KernelAbstractions.jl runs Julia kernels on the CPU through PoCL. A one-work-group launch through a command queue costs 2.5–5 µs, while the work-group function itself takes a few nanoseconds. That rules out small, frequent kernels (a 16×16 matmul, the many small loops of a PDE right-hand side), which Julia's own threading handles in 0.02–0.4 µs. With this extension, KA launches cost 0.05 µs on one work-group and 0.3–0.4 µs on eight threads, on par with Polyester.jl (JuliaGPU/KernelAbstractions.jl#906).

API (include/CL/cl_ext_pocl.h)

• clCreateHostKernelPOCL(kernel, device, work_dim, local_size, &err) prepares a kernel for one local size: it looks up the work-group function (specialized on the local size, generic in grid size and offset), runs the program-scope initializers and records the __local argument sizes. The handle retains the kernel.
• clRunHostKernelPOCL(hk, arg_values, num_groups, offset, first, count, scratch) runs work-groups [first, first + count) of the grid on the calling thread.
  • arg_values[i] points to argument i's value, as for clSetKernelArg; for pointer arguments, that value is the SVM/USM/device address.
  • scratch holds __local memory and the printf buffer; concurrent calls need distinct scratch.
  • The kernel runs in its own FP environment (round to nearest, FTZ per the build, exceptions masked) on x86 and AArch64; the caller's is restored, flags included.
  • Returns CL_OUT_OF_RESOURCES if a work-group failed.
• clGetHostKernelInfoPOCL (scratch size and alignment, refcount), clRetainHostKernelPOCL, clReleaseHostKernelPOCL.
• Contract: work-groups may run in any order, on any thread, concurrently, so they must not wait on each other. No image, sampler or pipe arguments; whole work-groups only.

Implemented for the in-process CPU drivers (basic, pthread, TBB) through three device ops; other drivers return CL_INVALID_OPERATION.

Commits

Based on upstream main, which has pocl#2410 (group ids as size_t).

1. Fix program-scope variable initialization on queried and rebuilt programs: Fix program-scope variable initialization on queried and rebuilt programs pocl/pocl#2407 (program-scope variable initialization from any thread), which host kernels depend on. It drops out on the next rebase once upstream merges it.
2. Add cl_pocl_host_kernel: the extension, docs and tests/runtime/test_cl_pocl_host_kernel.c (concurrent runs on one handle, locals and barriers, globals, offsets and partial ranges, FP environment, traps, lifetime, invalid uses, JIT).
3. Report the stack estimate as CL_KERNEL_PRIVATE_MEM_SIZE on CPUs: lets callers decide whether a work-group fits on their own stack.

Related: pocl#2408 (grid width). The JLLs carry the same work on upstream's 7.2 maintenance branch (pocl_next and pocl_standalone, JuliaPackaging/Yggdrasil#15308).

Results

Minimum µs per KernelAbstractions.jl launch, Ryzen 9 9950X, 8 threads. Queue = KA 0.10.1 on the release JLL.

	queue	host kernels
trivial kernel, ndrange 1	2.72	0.05
trivial kernel, 66²	4.61	0.37
naive matmul 16²	4.30	0.63
naive matmul 256²	2617	2292

Apple M1: ndrange 1 goes from 4.9 to 0.12 µs and 66² from 18.8 to 1.5 µs, but large kernels are 5–8% slower than through the queue. clRunHostKernelPOCL itself costs ~17 ns for one 128-item work-group. On x86, the generic-grid work-group function is within 2% of the queue's specialized one (single-threaded C benchmark, 64³–128³ stencils).

TODO before this is production-ready

PoCL:

• Shared work-group loop. pocl_cpu_run_host_kernel duplicates the pthread/TBB worker loop (argument setup, context, FP mode, printf, failure flag). Factor the loop, the FP enter/restore pair and the printf flush into common_utils.c and use it from all three; upstream would ask for this.
• Stack estimate is not an upper bound. It counts allocas, not the context arrays the work-item loops add for values live across barriers. Measure the work-group function's frame after codegen instead.
• Printf buffer in the scratch area. It takes the device's full printf buffer (16 MiB by default) per concurrent call; size it to zero for kernels that don't print.
• Fix program-scope variable initialization on queried and rebuilt programs pocl/pocl#2407 fast path. Queue launches of programs with globals now take the program lock on every launch; check readiness with an acquire load first.
• M1 large-kernel gap (5–8%): not investigated.
• Header: clarify that arg_values[i] for a pointer argument points to the address, not the address itself.
• Register the enum and error values (provisional now); decide whether to propose upstream.
• Optional: start pthread workers lazily, so host-kernel-only processes don't create an idle pool.

KernelAbstractions.jl (JuliaGPU/KernelAbstractions.jl#906), where most of the remaining work is:

• Oversubscription: the caller plus N team workers keep N+1 threads busy on N cores (-t N with an interactive thread), which makes long kernels 20–25% slower. Count the caller as one of the N, and claim chunks dynamically for every multi-chunk launch.
• One Julia thread: plain julia runs every kernel on one core, where the queue used all of them. Decide the policy (respect -t, or fall back to the queue for large grids).
• GC safety: decided from the previous launch's duration, so a first slow launch blocks GC on other threads. Default to GC-safe.
• Composability: launches from user-parallel code (@threads, @spawn) spawn a full team each; run serially on the caller when the pool is busy, as Polyester does.
• Compile latency: 120–145 ms per new kernel against 95–112 ms for KA 0.10.1 (and 41–48 ms for KA 0.9).
• Long-launch worker reuse (captured scope, interrupt handling) and yielding after short launches.

Testing

• Linux x86_64: the CPU tests pass on main plus this PR (ctest -LE cpu_fail), except 27 SPIR-V tests that also hang on main on this machine (system translator built for another LLVM).
• The 7.2 backport passes the new and affected tests, and KernelAbstractions.jl's full suite through both host kernels and the queue. Windows was validated through KA's KernelInterface tests (8633/8633) with a locally built MinGW JLL.