Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/agents/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958

A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount. This caused either `awf-agent` startup failure (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`) or, once worked around, an entrypoint abort at `mkdir -p /host$HOME/.m2` under `set -e`. **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup. `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`.

A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679.
A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679.

B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`.

Expand Down Expand Up @@ -254,7 +254,7 @@ Establish these facts before matching a failure mode:
| A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 |
| A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `<prefix>/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 |
| A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 |
| A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 |
| A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 |

## Category B — Self-hosted runners

Expand Down
5 changes: 5 additions & 0 deletions .github/aw/actions-lock.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@
"version": "1cf55bdc4f19e5f371b1e7fe888df649695ee48a",
"sha": "1cf55bdc4f19e5f371b1e7fe888df649695ee48a"
},
"github/gh-aw/actions/setup@b5db66ca43a7614d7d3efd248dfe7323f3080baa": {
"repo": "github/gh-aw/actions/setup",
"version": "b5db66ca43a7614d7d3efd248dfe7323f3080baa",
"sha": "b5db66ca43a7614d7d3efd248dfe7323f3080baa"
},
"github/gh-aw/actions/setup@v0.87.5": {
"repo": "github/gh-aw/actions/setup",
"version": "v0.87.5",
Expand Down
Loading
Loading