Skip to content

refactor: pruner engine, deploy matrix tags and shared test harness - #33

Merged
azlekov merged 3 commits into
mainfrom
refactor/pruner-engine
Oct 19, 2026
Merged

azlekov merged 3 commits into
mainfrom
refactor/pruner-engine

Conversation

@azlekov

@azlekov azlekov commented Oct 19, 2026

Copy link
Copy Markdown

Summary

Three cleanups deferred from #31 (/simplify review), one commit each, stacked on feat/runner-slots (base this PR on it, or retarget to main once #31 merges).

  1. pruner.sh engine — collect_units no longer forks per unit.
    • One find -printf per dir rule + one awk that aggregates by the glob's depth (a unit's size, newest file atime and bun's package.json atime are summed as the lines stream by; find lists a directory before its contents).
    • STORE_TABLE gets an opts column (shared, key=<file>) replacing the hard-coded actions exclusion and bun's unit_key; the rest of the per-store behaviour (rustup default protection, toolcache .complete markers, bun symlinks) is one store_hook dispatcher. The awk program is mawk-compatible (Ubuntu's default awk).
    • The find format is %P\t%y\t%A@\t%T@\t%s\t%d: %T@ is bun's old fallback key (unit mtime when there is no package.json), %d is the depth.
  2. CI — Dockerfile versions are read once in each workflow's matrix job and exposed as outputs. deploy.yml's matrix jq emits each entry's own tag list (merge) and image tag (build), replacing the per-line enable=${{ matrix.variant == ... }} and the identity metadata step. The variant/target/platform/cache-scope table is .github/jq/matrix.jq, shared by build.yml and deploy.yml via jq -L; the tag rules are .github/jq/tags.jq.
  3. Tests — tests/harness.sh (scratch dir + trap, fail/gone/kept/finish, at_exit) and tests/lib.sh (pruner fixtures on top, so tests/pruner-*.sh are unchanged). slots.sh and image-smoke.sh source the harness; image-smoke.sh runs its checks with docker exec in one idle container instead of ~13 docker runs.

Benchmark: one pruner pass over a synthetic 100k-unit cache

tests/bench/pruner-100k.sh (50k bun packages, 30k Gradle module versions, 20k npm cache files; budget at 90%, so ~10% is evicted). ubuntu:24.04 container, Docker on Apple silicon (arm64):

before after
collect_units (100k) 112.9 s 2.9 s
full pass 129.9 s 13.5 s
units / evicted 100000 / 9836 100000 / 9836

The remaining ~10 s is the evict loop (rm -rf + hooks per evicted unit) and the sort; not touched here.

Verification

  • tests/pruner-*.sh and tests/slots.sh pass unchanged (ubuntu:24.04); tests/image-smoke.sh passes on local std/slim/ndk images and fails with exit 1 on a wrong JDK, leaving no container behind.
  • Old vs new collect_units on a fixture with every store (hidden dirs, symlinks, names with spaces, rustup default, no-package.json bun unit): the same 34 records (key, size, store, stop, path).
  • deploy.yml: the generated build/merge matrices and tag lists were diffed against the previous workflow's (its enable= conditions evaluated per entry): 20 build entries and 10 merge entries identical, including each image tag. build.yml's PR matrix is identical (key order aside).
  • actionlint and shellcheck -x clean.

🤖 Generated with Claude Code

@azlekov
azlekov added this pull request to stack #34 October 19, 2026 19:54
Base automatically changed from feat/runner-slots to main October 19, 2026 19:55
azlekov and others added 3 commits October 19, 2026 22:55
collect_units forked a find|awk pair plus stat subshells per evictable unit, tens of
thousands of times on Gradle/bun caches. It now runs one find -printf per dir rule and
one awk that sums size and newest file atime per unit (units sit at the glob's depth and
find lists a directory before its contents), taking bun's package.json atime from the
same stream.

STORE_TABLE gets an opts column (shared, key=<file>) replacing the hard-coded actions
exclusion and the bun unit_key override; the remaining per-store behaviour (rustup default
protection, toolcache .complete markers, bun symlinks) sits in one store_hook dispatcher.

Adds tests/bench/pruner-100k.sh, a timing of one pass over ~100k synthetic units.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… matrix jq

The matrix job extracts the Dockerfile versions once and exposes runner_tag as an output;
the build and merge jobs of deploy.yml and the build job of build.yml no longer repeat the
extraction. deploy.yml's matrix entries now carry their own tag list (merge) and the tag
baked into the image (build), replacing the per-line enable=${{ matrix.variant == ... }}
and the image-identity metadata step.

The variant/target/platform/cache-scope table moves to .github/jq/matrix.jq, shared by
build.yml and deploy.yml via jq -L; the tag rules are in .github/jq/tags.jq. Generated
tags and PR matrix are identical to the previous ones.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tests

tests/harness.sh holds the pure harness (scratch dir with exit trap, fail/gone/kept/finish,
at_exit); tests/lib.sh keeps only the pruner fixtures on top of it, so tests/pruner-*.sh
are unchanged. slots.sh and image-smoke.sh source the harness instead of keeping their own
fail counters. image-smoke.sh runs its checks with docker exec in one idle container
instead of a docker run per check. CI skips harness.sh and lib.sh in the test loop and
shellchecks tests/bench.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@azlekov
azlekov force-pushed the refactor/pruner-engine branch from 2586dea to 7e0c21a Compare October 19, 2026 19:55
@azlekov
azlekov merged commit 385578c into main Oct 19, 2026
@azlekov
azlekov deleted the refactor/pruner-engine branch October 19, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant