[maleadt]

Implements the first proposal of #83: in foreach_range(...; schedule = :steal), thieves prefer victims that share their last-level cache, then their NUMA node, and steal from other nodes only once a pass over their own node found nothing. Stacked on #84, whose shared rescue clock this builds on; review only the last two commits.

Design

• Topology comes from Linux sysfs, read once in Pool(): the sharing set of each CPU's highest-level unified/data cache, the node CPU lists (ignored when they overlap, as with the emulated NUMA nodes of a Raspberry Pi 5), and node distances. It's only used when every worker's affinity mask is a single CPU of its own, which Pool() already reads (Julia ≥ 1.11). Otherwise, on other OSes, on 1.10, or when any file is missing, the order stays cyclic. We read sysfs instead of using SysInfo.jl/Hwloc.jl: SysInfo, which ThreadPinning uses, exposes no cache domains or node distances, and both load Hwloc (~50–250 ms) for every user; the sysfs reads take ~0.6 ms, and only for pinned pools.
• Victim order, per worker, immutable in RangeState: LLC ∩ node → rest of the node → remote nodes by distance; within a tier, rotated after the worker (remote tiers by the thief's rank within its node, so a node's thieves don't all start on the same remote victim). Victims beyond p are skipped.
• Crossing rule: thieves repeat local passes while they find work, go remote only after a fruitless local pass, and return to their node right after a remote steal (once its republished remainder is drained). Retiring still requires a full fruitless scan; quarter steals, startup protection and PROTECTED_PASSES are unchanged.
• Proximity is the executing worker's: a block the caller rescues uses the caller's order (cyclic off the pool), while republished work still goes to the rescued block.
• Contract: pin before the first parallel loop, or call ThreadTeams.shutdown!() after pinning; documented in the docstring and caveats page. No revalidation, no public API. The static path is unchanged.

Left out after measuring: publishing commands round-robin over nodes (proposal 3's cheap variant) gave no smaller start spread and was slower on desktops; the go-word protocol it would really need is a bigger change.

Results

Paired ratios, local-first / cyclic, measured in the same process with randomized interleaving (median over 3 processes), with #84 underneath; < 1 is faster. Forced :steal foreach_range, threads pinned with ThreadPinning.

2× Xeon Gold 6430 (L3 = socket = node), Julia 1.11:

threads, pinning	axpy 1e5	axpy 1e6	axpy 1e7	stencil 126³ / 128³
32 :numa	0.85	0.87	0.96	1.00 / 1.01
64 :cores	1.01	0.89	0.90	0.94 / 0.92
64 :numa	0.87	0.87	0.89	0.79 / 0.82
32 :cores (one node: topology unused)	1.00–1.05

Indices moved across sockets by steals drop from 37–81% (cyclic) to 0–4% on axpy. KernelAbstractions' CPU back-end (JuliaGPU/KernelAbstractions.jl#906) forced to steal follows (0.80–0.88 at :numa); its adaptive policy gets 0.72–0.74 on axpy 1e7 at 64 threads, because faster launches keep it under its 50 µs steal threshold.

One node, two L3s (Ryzen 5950X / 9950X, 16 threads): axpy 1e5 0.85 / 0.95, axpy 1e6 1.02 / 0.95; stencils and an imbalanced triangular loop within ±2%. Stolen indices that stay within the CCD: 31% → 89% (5950X, axpy 1e5).

Hosts where topology is unused (96-core Ampere Altra with no shared cache in sysfs, macOS, unpinned): same code as cyclic, no change beyond noise.

Topology detection was checked against lscpu/sysfs on a Raspberry Pi 5 (overlapping nodes rejected), Jetson Orin/Xavier/Nano, an Intel laptop, an M1 Mac (cyclic) and the hosts above. Building the orders takes 0.6–1.3 ms at pool start for pinned pools, ~0.03 ms otherwise.

Tests: pure order construction on synthetic topologies (2 nodes × 2 LLCs with :cores/:numa layouts, sub-NUMA clustering, overlapping nodes, missing caches, p < nworkers), the sysfs parser on fake trees, and steal behaviour with installed orders (a thief takes a same-node victim before a remote one, and returns locally after a remote steal); these fail with cyclic order. Full suite on Julia 1.12 (-t 16,1) and 1.10 (-t 8).

[maleadt]

The first Windows run was cancelled after hitting the 60-minute limit with no test output; a rerun passed in 8 minutes. The same kind of hang shows up on earlier runs without these changes (the tb/foreach-range Windows job and main's Julia min - 4 threads job on 2026-10-26). A likely candidate is a race in the "interrupted while waiting for the workers" test, which also fails on main (about 1 in 40 runs with julia +1.10 -t 16): its observer can schedule the main task a second time while it is still queued (val already in a list).

[github-actions]

Benchmark Results (Julia v1)

Time benchmarks

	main	7c63ba6...	main / 7c63ba6...
ad/forward tforeach team, 1000	2.25 ± 0.32 μs	2.2 ± 0.34 μs	1.02 ± 0.22
ad/forward tforeach team, 100000	0.0626 ± 0.16 ms	0.0617 ± 0.15 ms	1.01 ± 3.6
ad/reverse serial loop, 1000	11.2 ± 0.35 μs	10.8 ± 0.34 μs	1.04 ± 0.046
ad/reverse serial loop, 100000	1.03 ± 0.64 ms	0.996 ± 0.63 ms	1.04 ± 0.92
ad/reverse tforeach team, 1000	25.5 ± 0.91 μs	24.1 ± 0.88 μs	1.06 ± 0.054
ad/reverse tforeach team, 100000	1.23 ± 0.66 ms	1.18 ± 0.75 ms	1.04 ± 0.86
ad/reverse tmapreduce team, 1000	11.7 ± 0.55 μs	11.6 ± 0.45 μs	1.01 ± 0.061
ad/reverse tmapreduce team, 100000	0.198 ± 0.24 ms	0.187 ± 0.054 ms	1.06 ± 1.3
fastbroadcast/9 arguments serial, 100x100x8	0.073 ± 0.008 ms	0.0727 ± 0.0084 ms	1 ± 0.16
fastbroadcast/9 arguments thread, 100x100x8	0.066 ± 0.0078 ms	0.065 ± 0.0092 ms	1.02 ± 0.19
fastbroadcast/log Base, 1000	3.71 ± 0.081 μs	3.52 ± 0.082 μs	1.05 ± 0.034
fastbroadcast/log Base, 100000	0.36 ± 0.01 ms	0.355 ± 0.01 ms	1.01 ± 0.041
fastbroadcast/log serial, 1000	3.64 ± 0.091 μs	3.54 ± 0.081 μs	1.03 ± 0.035
fastbroadcast/log serial, 100000	0.346 ± 0.0099 ms	0.354 ± 0.012 ms	0.978 ± 0.043
fastbroadcast/log thread, 1000	2.2 ± 0.07 μs	2.09 ± 0.07 μs	1.05 ± 0.049
fastbroadcast/log thread, 100000	0.17 ± 0.011 ms	0.171 ± 0.012 ms	0.996 ± 0.094
overhead/@tasks team	0.3 ± 0.041 μs	0.311 ± 0.041 μs	0.965 ± 0.18
overhead/OhMyThreads tforeach	2.03 ± 0.24 μs	2.03 ± 0.18 μs	1 ± 0.15
overhead/Threads.@threads :dynamic	2.24 ± 0.86 μs	2.21 ± 0.92 μs	1.01 ± 0.57
overhead/serial loop	20 ± 10 ns	20 ± 10 ns	1 ± 0.71
overhead/tforeach greedy team	0.501 ± 0.06 μs	0.571 ± 0.06 μs	0.877 ± 0.14
overhead/tforeach team	0.3 ± 0.041 μs	0.281 ± 0.03 μs	1.07 ± 0.19
overhead/tmapreduce team	0.35 ± 0.04 μs	0.36 ± 0.04 μs	0.972 ± 0.15
trixi/reset du Threads.@threads :static	0.114 ± 0.0043 ms	0.114 ± 0.0073 ms	1 ± 0.074
trixi/reset du serial	1 ± 0.029 μs	0.912 ± 0.01 μs	1.1 ± 0.034
trixi/reset du team	0.942 ± 0.25 μs	1.01 ± 0.22 μs	0.931 ± 0.32
trixi/reset du team, 10 in a row	7.29 ± 0.54 μs	7.52 ± 0.33 μs	0.969 ± 0.084
work/serial, 1000	1.75 ± 0.01 μs	1.67 ± 0.01 μs	1.05 ± 0.0087
work/serial, 100000	0.173 ± 0.0074 ms	0.173 ± 0.0039 ms	0.999 ± 0.049
work/tforeach team, 1000	1.19 ± 0.04 μs	1.11 ± 0.03 μs	1.07 ± 0.046
work/tforeach team, 100000	0.0868 ± 0.0095 ms	0.0839 ± 0.012 ms	1.03 ± 0.18
time_to_load	0.0678 ± 0.0023 s	0.071 ± 0.0025 s	0.955 ± 0.046