Make ClockCache hits lock-free

CLAUDE.md

@@ -44,7 +44,11 @@ julia --project=/tmp/runic -e 'using Pkg; Pkg.add("Runic"); using Runic; Runic.m
 - `src/registry.jl`: global caches, one untyped `C{Any,Any}` per function (its budget), looked up
   lock-free in an atomically published `IdDict` snapshot.
 - `src/containers/`: `LRU` and `ClockCache`, sharing the `AbstractCache` interface. Keys are
-  stored as `Key{Any}` and probed with a concretely typed `Key{K}`, so hits never box.
+  stored as `Key{Any}` (or `Any` plus hash) and probed with a concretely typed `Key{K}`, so hits
+  never box. `LRU` takes its lock for every operation; `ClockCache` (`clock.jl`) has lock-free
+  hits, through an atomically published open-addressing table of immutable entries, and keeps
+  the three invariants listed in its source (`research/lockfree-clock.md`). Task-local caches
+  use `LocalClockCache` (`localclock.jl`), the locked `ClockCache`, via `_localtype` in `call.jl`.
 - `src/preferences.jl`: default settings from Preferences.jl, resolved once per function
   (runtime > function > package > `[Cached]` > built-in), and `set_cache_preferences!`.
 - `src/api.jl`: `cache_info`, `empty_caches!`, `set_cache_size!`, and the `cache_dashboard` and

benchmark/Project.toml

@@ -4,6 +4,7 @@ Cached = "1b238080-9255-4fe9-b224-89eb24efe93b"
 LRUCache = "8ac3fa9e-de4c-5943-b1dc-09c6b5f20637"
 Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 
 [sources]
 Cached = {path = ".."}

benchmark/contention.jl

@@ -0,0 +1,134 @@
+# Cost of a lookup when all threads share one cache, against task-local caches.
+# "locked Clock" is the `ClockCache` of task-local caches (`Cached.LocalClockCache`), whose hits
+# take the lock, shared here for comparison.
+#
+#     for t in 1 2 4 8; do julia --heap-size-hint=4G --project=benchmark -t $t benchmark/contention.jl; done
+#
+# One task per thread, each doing `M` lookups from its own key sequence. Reports the median over
+# `REPS` runs of the time per lookup within a task (ns/lookup, lower is better; perfect scaling
+# keeps it constant as threads are added), and the bytes allocated per lookup.
+
+using Random, Printf, Statistics
+using Cached: Cached, ClockCache, LocalClockCache, LRU, @cached, GlobalCache, TaskLocalCache
+
+const M = 200_000
+const REPS = 7
+
+const CONTAINERS = [
+    "ClockCache" => n -> ClockCache{Any, Any}(; maxsize = n),
+    "locked Clock" => n -> LocalClockCache{Any, Any}(; maxsize = n),
+    "LRU" => n -> LRU{Any, Any}(; maxsize = n),
+    "task-local Clock" => nothing, # one locked Clock per task, uncontended
+]
+
+function zipf(rng, n, s, len)
+    cdf = cumsum(1 ./ (1:n) .^ s)
+    cdf ./= cdf[end]
+    return [searchsortedfirst(cdf, rand(rng)) for _ in 1:len]
+end
+
+tuplekey(i) = (i, i % 7, :leg, Float64(i))
+compute(k::Int) = 3k
+compute(k::Tuple) = k[1]
+
+# name => (key sequence per task, cache capacity, prefill keys)
+function workloads(ntasks)
+    rngs = [Xoshiro(t) for t in 1:ntasks]
+    return [
+        "hits, Int keys (1k)" => ([rand(r, 1:1_000, M) for r in rngs], 10_000, 1:1_000),
+        "hits, 4-tuple keys (1k)" => ([tuplekey.(rand(r, 1:1_000, M)) for r in rngs], 10_000, tuplekey.(1:1_000)),
+        "hits, one hot Int key" => ([fill(7, M) for r in rngs], 10_000, 7:7),
+        "zipf 1.0 (100k keys, cap 10k)" => ([zipf(r, 100_000, 1.0, M) for r in rngs], 10_000, 1:0),
+        "uniform (20k keys, cap 10k)" => ([rand(r, 1:20_000, M) for r in rngs], 10_000, 1:0),
+    ]
+end
+
+function run!(c, seq)
+    acc = 0
+    for k in seq
+        acc += get!(() -> compute(k), c, k)::Int
+    end
+    return acc
+end
+
+# ns per lookup in each task, and total bytes allocated
+function threaded(make, seqs, cap, prefill)
+    shared = make === nothing ? nothing : make(cap)
+    caches = [shared === nothing ? LocalClockCache{Any, Any}(; maxsize = cap) : shared for _ in seqs]
+    for c in unique(caches), k in prefill
+        get!(() -> compute(k), c, k)
+    end
+    times = zeros(length(seqs))
+    start = Threads.Atomic{Int}(0)
+    bytes = @allocated @sync for (t, seq) in enumerate(seqs)
+        Threads.@spawn begin
+            Threads.atomic_add!(start, 1)
+            while start[] < length(seqs) # start together
+                ccall(:jl_cpu_pause, Cvoid, ())
+            end
+            t0 = time_ns()
+            run!(caches[t], seq)
+            times[t] = (time_ns() - t0) / length(seq)
+        end
+    end
+    return mean(times), bytes / sum(length, seqs)
+end
+
+# ns per hit through `@cached` functions using each cache style, Int and 4-tuple keys
+@cached f_clock(x) = compute(x)
+@cached f_locked(x) = compute(x)
+@cached f_lru(x) = compute(x)
+@cached f_local(x) = compute(x)
+Cached.CacheStyle(::typeof(f_clock), x) = GlobalCache{ClockCache}()
+Cached.CacheStyle(::typeof(f_locked), x) = GlobalCache{LocalClockCache}()
+Cached.CacheStyle(::typeof(f_lru), x) = GlobalCache{LRU}()
+Cached.CacheStyle(::typeof(f_local), x) = TaskLocalCache{ClockCache}()
+const CACHED = ["ClockCache" => f_clock, "locked Clock" => f_locked, "LRU" => f_lru, "task-local Clock" => f_local]
+
+function callall(f::F, seq) where {F}
+    acc = 0
+    for k in seq
+        acc += f(k)::Int
+    end
+    return acc
+end
+
+function threaded_cached(f, seqs, keys)
+    times = zeros(length(seqs))
+    bytes = @allocated @sync for (t, seq) in enumerate(seqs)
+        Threads.@spawn begin
+            callall(f, keys) # fill this task's cache (task-local)
+            t0 = time_ns()
+            callall(f, seq)
+            times[t] = (time_ns() - t0) / length(seq)
+        end
+    end
+    return mean(times), bytes / sum(length, seqs)
+end
+
+nt = Threads.nthreads()
+println("Julia ", VERSION, ", ", nt, " threads (", nt, " tasks), ", M, " lookups per task, median of ", REPS, " runs")
+println("load average: ", read(`cat /proc/loadavg`, String))
+@printf("%-30s %-17s %10s %10s %8s\n", "workload", "container", "ns/lookup", "(min)", "B/lookup")
+for (wname, (seqs, cap, prefill)) in workloads(nt)
+    for (cname, make) in CONTAINERS
+        threaded(make, seqs, cap, prefill) # compile
+        rs = [threaded(make, seqs, cap, prefill) for _ in 1:REPS]
+        @printf(
+            "%-30s %-17s %10.1f %10.1f %8.2f\n", wname, cname,
+            median(first.(rs)), minimum(first.(rs)), median(last.(rs))
+        )
+    end
+end
+for (kname, keys) in ("@cached hits, Int" => 1:1_000, "@cached hits, 4-tuple" => tuplekey.(1:1_000))
+    seqs = [rand(Xoshiro(t), keys, M) for t in 1:nt]
+    for (cname, f) in CACHED
+        foreach(f, keys)
+        threaded_cached(f, seqs, keys)
+        rs = [threaded_cached(f, seqs, keys) for _ in 1:REPS]
+        @printf(
+            "%-30s %-17s %10.1f %10.1f %8.2f\n", kname, cname,
+            median(first.(rs)), minimum(first.(rs)), median(last.(rs))
+        )
+    end
+end

research/design.md

@@ -77,18 +77,20 @@ Cached provides its own containers, so it no longer depends on LRUCache.jl.
 Both implement the same small interface: `get!`, `get`, `haskey`, `empty!`, `resize!`, `length`, and hit/miss statistics.
 
 - **`LRU{K,V}`** is array-backed. It uses a `Dict` index, slots stored in vectors, and `prev`/`next` stored as integer vectors. Nodes are never allocated, and eviction is exact LRU.
-- **The index is a `Dict{Key{Any},Int}`.** Each stored `Key{Any}` holds the key and its hash. Lookups probe it with a concretely typed `Key{K}`, whose `isequal` checks `s.key isa K` before comparing, so the comparison is static and the key is never boxed. A `Key{Any}` cannot serve as the probe: storing the key in an `Any` field boxes it (96 B per lookup on TensorKit-like keys).
+- **The index of `LRU` is a `Dict{Key{Any},Int}`.** Each stored `Key{Any}` holds the key and its hash. Lookups probe it with a concretely typed `Key{K}`, whose `isequal` checks `s.key isa K` before comparing, so the comparison is static and the key is never boxed. A `Key{Any}` cannot serve as the probe: storing the key in an `Any` field boxes it (96 B per lookup on TensorKit-like keys).
   - This holds for `C{Any,Any}` as well, where a plain `Dict{Any,…}` would allocate on every hit.
   - Eviction never re-hashes, because the hash is stored.
   - Keys of different types are different entries, even when `isequal` (`f(3)` and `f(3.0)`). Lookups on a typed cache convert the key to `K` first.
-- **`ClockCache{K,V}`** uses second-chance eviction: a ring of slots with a reference bit. A hit only sets the bit and never reorders the ring, which makes it cheap for read-heavy shared caches.
-- Each container takes a size limit, either a **count** or **bytes** measured by a `by` function. Each one holds its own lock. Task-local containers skip the lock.
+- **`ClockCache{K,V}`** uses second-chance eviction: a ring of entries with a reference bit. A hit only sets the bit (if it is clear) and never reorders the ring, so **hits take no lock** (see `lockfree-clock.md`).
+  - Its index is a fixed-size open-addressing table of immutable entries (each holding the key as `Any` with its hash, probed with a `Key{K}` as above), replaced as a whole when it fills up and published atomically. Readers load it and its slots with acquire ordering; every mutation (miss, insertion, eviction, `delete!`, `resize!`, `empty!`) still takes the lock.
+  - Hits and misses are counted per thread, in separate cache lines, so hits write no shared memory.
+  - A lookup racing with a removal may return the removed entry; the value always belongs to its key.
+- Each container takes a size limit, either a **count** or **bytes** measured by a `by` function, and holds its own lock.
+- **Task-local caches** of the default container use `LocalClockCache`, an internal `ClockCache` with the `Dict` index above and locked hits (uncontended, so cheap), which saves the per-thread counters (64 B per thread) in every task's cache. It evicts in the same order.
 - **Default for `GlobalCache`: `ClockCache`.**
-  - On synthetic workloads (`benchmark/containers.jl`), it beat `LRU` everywhere single-threaded (28 vs 32 ns for an all-hit lookup) and had a slightly better hit rate under Zipf access. Both were 1.5–3× faster than LRUCache.jl.
-  - On the TensorKit and SUNRepresentations integration, the two were indistinguishable.
-  - It stays the default because a hit only sets a bit, which makes lock-free hits possible.
-- **open**: contention. With 8 threads, every container's per-lookup cost *rises* (all hits: about 50–75 ns, against 28–32 ns single-threaded), because every lookup takes the cache's single lock.
-  Sharding by key hash, or lock-free reads for `ClockCache`, are the candidate fixes. `TaskLocalCache` is the workaround meanwhile.
+  - On synthetic workloads (`benchmark/containers.jl`), the locked `ClockCache` beat `LRU` everywhere single-threaded (28 vs 32 ns for an all-hit lookup) and had a slightly better hit rate under Zipf access. Both were 1.5–3× faster than LRUCache.jl. On the TensorKit and SUNRepresentations integration, the two were indistinguishable.
+  - With lock-free hits it is about 2× faster again single-threaded (14 vs 31 ns, `Int` keys), and shared hits scale: about 25 ns at 8 threads, against about 500 ns with a lock (`benchmark/contention.jl`). That is faster than task-local caches.
+- **open**: contended misses. Misses still serialize on the cache's lock, so miss-heavy shared workloads (all threads computing new keys) scale poorly; `TaskLocalCache` or a sharded cache would help there (see `lockfree-clock.md`). `LRU` hits still take the lock.
 
 ## Limits