Skip to content

Commit 1876901

Browse files
authored
Document AWF A21 helper-staging fix (#7734)
* Initial plan * docs: update runner doctor A21 references * docs: correct A21 implementation citation --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent 928d4ae commit 1876901

4 files changed

Lines changed: 10 additions & 8 deletions

File tree

.github/agents/self-hosted-runner-doctor.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ Prefer the narrowest match. Examples:
113113
- `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658)
114114
- `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959)
115115
- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)
116-
- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21
116+
- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679)
117117

118118
### 4. Check for known gaps and notable fixes
119119

@@ -131,7 +131,7 @@ A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958
131131

132132
A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount. This caused either `awf-agent` startup failure (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`) or, once worked around, an entrypoint abort at `mkdir -p /host$HOME/.m2` under `set -e`. **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup. `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`.
133133

134-
A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`, helper staging moved to `/run/awf-lib` as related hardening, nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`, and `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection. Legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images. Upgrade AWF to include github/gh-aw-firewall#7679.
134+
A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679.
135135

136136
B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`.
137137

@@ -254,7 +254,7 @@ Establish these facts before matching a failure mode:
254254
| A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 |
255255
| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `<prefix>/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 |
256256
| A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 |
257-
| A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; helper staging moved to `/run/awf-lib` as related hardening; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 |
257+
| A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 |
258258

259259
## Category B — Self-hosted runners
260260

0 commit comments

Comments
 (0)