[azlekov]

One PR for the runner refactor. It simplifies the setup, adapts it to each host, and keeps caches healthy on every machine in the fleet: Raspberry Pi 5, Hetzner and beelink-class hosts. It merges the 10 upstream compscidr commits, plus four work packages built in parallel sessions.

Closes #7
Closes #8
Closes #15
Closes #16
Closes #17
Closes #19

It prepares for #18 (emulator): hosts/sizing.sh reports /dev/kvm, and the emulator itself is a follow-up.

What changed

Runtime topology (W3)

• One runner service scaled by RUNNERS (plus CONTROL_RUNNERS) replaces three hand-copied services and 36 named volumes.
• Each container flock-claims RUNNER_DATA_DIR/slots/<id> for its lifetime. The work dir and all caches live in the slot, and the tools' default paths symlink into it.
• The data dir is bind-mounted at the same host path, so docker run -v "$PWD:…", container actions and container: jobs see the checkout (Container actions and docker run -v get empty mounts: runner work dir must be mounted at the same host path #19). Verified from inside a runner container.
• Optional RUNNER_CPUS / RUNNER_MEMORY per runner. Under a CPU quota the preflight caps Gradle (org.gradle.workers.max), cargo, make and cmake parallelism.
• hosts/sizing.sh light|large suggests RUNNERS, limits, the cache budget, and :slim on arm64.
• pre-job.sh marks the slot busy, waiting on the pruner's lock first. cleanup.sh clears busy and wipes /tmp flat; the recursive spare-the-workdir walk is gone.

Pruner (W4)

• One lru_entries engine and a store table replace three bespoke functions. It evicts whole entries wherever partial ones break builds: bun packages, Gradle versions, rustup toolchains, toolcache versions, the pnpm store, and so on.
• A per-host budget, min(CACHE_BUDGET_BYTES, CACHE_BUDGET_PCT), is split across slots, with global LRU within a slot.
• Busy slots are skipped. Runs are adaptive: an early pass and a tighter docker prune when free disk drops below PRUNE_FREE_PCT.

CI (W2)

• JDK matrix cut to LTS + latest (17, 21, 25, 27) and Node 22, 24.
• type=gha layer cache scoped per platform, timeout-minutes, and zstd-compressed layers.
• Variants: slim / ndk tags. An image smoke test runs on PRs.
• shellcheck covers every script, and docker compose config is validated.
• check-sdk-updates opens PRs with the org App token (needs APP_ID / APP_PRIVATE_KEY repo secrets) and auto-merges. Non-LTS JDKs are dropped when a newer GA lands.

Image (W1)

• Stages: sdk (fixed JDK, so the SDK layer is identical for every tag), core (base, one apt layer incl. xxd/pigz, JDK, Node, hooks), then runner-slim = core, android = core + SDK, runner-ndk = android + NDK/cmake, and runner = android (the last, default stage).
• rustup without a toolchain: the image ships a checksum-verified rustup-init, and the preflight sets rustup up in each slot's ~/.cargo. Toolchains a job installs persist in the slot (stable --profile minimal: 7 s cold, 0 s warm).
• Platform 37: android-37.x is stable (empty <codename/>); the old preview filter skipped it. Upstream's fix and the 37.2 bump come in through the sync.

Upstream sync

• compscidr/main: platform 37.2, empty-<codename>-is-stable SDK detection, and build-tools / NDK spelled-out tags.
• The NDK-version tags now go on the ndk variant, the only one that ships an NDK.

Docs

• README rewritten as a guided setup: GitHub App, sizing, variants and tags, how slots work, workflow tips, caches, maintenance, config reference, troubleshooting and upgrade steps.

Commits

1. chore: merge upstream compscidr/main. This is a real merge, so later upstream syncs don't re-conflict.
2. refactor(pruner): the store-table engine, per-slot budgets, and busy-slot safety.
3. feat(compose): scalable slots, identical-path work dirs, sizing, and the job hooks.
4. feat(image): the slim, default and ndk targets, the shared SDK layer, xxd and rustup.
5. ci: the matrix, the layer cache, variants, the smoke test, and App-token SDK PRs.
6. docs(readme).

Figures

	Before	After
Compose runner services / named volumes / lines	3 + 2 control / 36 / 220	1 + 1 scaled / 1 / 129
YAML lines to add a runner	~30 in 3 places	0 (RUNNERS=N)
Stores bounded by the pruner	5 of 11	13 of 13 (+ pub, hosted tool cache)
Pruner LRU implementations	3	1 + table
Deploy jobs (builds + merges)	32 + 16	20 + 10, of which 16 + 8 are the main images (−50 %) and the rest add slim + ndk
PR build legs	11	8, including slim / ndk smoke legs, all layer-cached
docker run -v / container actions on self-hosted	broken	working
Default image (amd64, uncompressed)	5.65 GiB, with NDK + cmake	3.18 GiB (−44 %); ndk 5.42 GiB opt-in
Small-host image	none	slim: 2.81 GiB amd64 / 2.58 GiB arm64, no Android SDK
Weekly SDK bump on a host	re-pull every layer above the SDK	re-pull the 436 MB SDK layer only
Job-completed hook	35 s median before #16's fix	0 s (measured 10-16 to 10-19 over 960 jobs)

Baselines were measured over 1,068 self-hosted jobs in 6 onboarded repos, about 1,900 jobs a week fleet-wide. Image sizes are from local amd64/arm64 builds; all three targets pass tests/image-smoke.sh.

What the onboarded repos get

• bun / Next.js (nis-suite, dirahq-cloud, dirahq-landing, kodni.me, infrasensing-senslogging): no repeat of the 10-18 server-only eviction break. npm and pnpm are bounded, and container actions work.
• Android / Gradle (nis-inventory-android, cm30, infrasensing-mobile, kodni-mobile, ttlock-android): platform 37.2 is baked in. Gradle cache eviction removes whole versions. Gradle workers are capped per runner.
• Flutter (smart-irrigation): the pub cache and the Flutter SDK in the hosted tool cache now persist.
• Rust (any private Rust repo moving to self-hosted): rustup is ready, and toolchains stay warm per runner.
• Small hosts (RPi 5): a slim image and a cache budget that scales with the disk.

Upgrading a host (breaking)

git pull && docker compose down
# .env: RUNNERS=N (hosts/sizing.sh), CONTROL_RUNNERS=2 if COMPOSE_PROFILES=control was used
docker compose pull && docker compose up -d

• Caches start cold once.
• Old named volumes can be removed afterwards (README → Upgrading from named volumes).
• COMPOSE_PROFILES=control and hosts/no-caches.compose.yml are replaced by CONTROL_RUNNERS and RUNNER_CACHES=off.

Verification

• tests/*.sh pass in ubuntu:24.04: slots, pruner budget, busy slots, and bun, cargo, gradle, pnpm, rustup and toolcache eviction.
• shellcheck and actionlint are clean, and docker compose config passes for RUNNERS=1/3/5.
• A stub image under compose (RUNNERS=2, plus a control runner) checked that:
  • each runner gets a distinct slot, kept across restarts and when scaling to 3;
  • per-slot labels come from .env;
  • the Container actions and docker run -v get empty mounts: runner work dir must be mounted at the same host path #19 probe passes on both slots;
  • the hooks set and clear busy;
  • CPU-quota detection works under --cpus 1.5.
• runner, runner-slim (amd64 + arm64) and runner-ndk were built and smoke-tested locally.
• Still to do after merge: a first deploy and a one-host rollout, to re-measure pull time and cache footprint.

Follow-ups (other repos)

• dodi-smart/.github: lift the container-action ban in AGENTS.md.
• setup-stack: drop the job-scoped PUB_CACHE and the ARM64 $RUNNER_TEMP cache isolation.
• dev-standards stacks.md: fix the "caches are job-scoped" line.

🤖 Generated with Claude Code