Skip to content

Add a skill for verifying Dependabot PRs, and check the dev server in CI - #142

Open
conatus wants to merge 6 commits into
masterfrom
claude/dependabot-integration-tests-1n5kl6
Open

conatus wants to merge 6 commits into
masterfrom
claude/dependabot-integration-tests-1n5kl6

Conversation

@conatus

@conatus conatus commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Dependabot PRs pile up because we can't tell from CI alone whether a bump is safe: our test coverage doesn't reliably touch the code that uses each library. This adds a routine that builds that coverage one package at a time and fixes what upgrades break, so bumps reach a person ready to merge and can eventually merge themselves.

What's in it

/verify-dependabot skill (.claude/skills/verify-dependabot/SKILL.md). For a Dependabot PR it:

  1. Finds where the bumped package touches our code. Most of our Dependabot PRs are transitive security bumps that we never import, so it traces them with npm why/composer why and classifies them: runtime (direct or transitive), build tooling, dev server, or test tooling.
  2. Writes the smallest test of our use of the package (never of the library itself). For build and dev-server tooling, the build and the dev server are the test.
  3. Proves the test can fail: it passes on the old version, fails when our code is broken, fails when the library's code in node_modules/vendor is changed, and passes on the new version.
  4. Pushes the test plus a ledger entry to the Dependabot branch.
  5. Fixes what breaks. If the upgrade breaks a new test, an existing test, the build, lint or E2E, it fixes our code until CI is green. It never skips or weakens a test, and never pins the package back.
  6. Gets its commits reviewed. Once CI is green, a review agent with fresh context runs the code-review skill on the PR and posts its findings as inline review comments. The skill fixes what's reasonable and answers each thread: fixed threads get the commit and are resolved, declined ones get the reason and stay open. Then a new agent reviews again, until a reviewer is satisfied. After three rounds it stops and asks the person below for help. A status comment on the PR records the round, so the daily run resumes where it stopped.
  7. Moves through gears:
    • First time a package is seen: once CI is green and a review agent is satisfied, it posts the evidence and requests review from whoever added the dependency (or the top-level dependency that pulls it in), falling back to whoever last changed it.
    • After that, minor and patch bumps: auto-merge on green CI, as long as the package isn't used anywhere new. If CI is red, it fixes it and the PR goes back to a person.
    • Major bumps (and 0.x minor bumps): always go to a person.

Ledger (.github/dependency-coverage/): one JSON file per package, recording its class, where we use it and which tests verify it. A file only reaches master through a reviewed Dependabot PR, so its presence on master is the approval record. One file per package means Dependabot PRs open at the same time never conflict.

Dev server check in CI (.github/workflows/dev-server.yml, scripts/dev-server-smoke.sh). Designers, engineers and headless Claude Code sessions all rely on the dev server, and nothing in CI ran it. The script starts it and checks the page, the hot-reloading bundle and the stubbed /join/v1/* endpoints. It fails when I add a syntax error to the source or change the stub route. Run it locally with npm --prefix packages/join-flow run test:dev-server.

scripts/bundled-packages.js tells the skill whether a package ships in the production bundle or is build tooling only.

First run

The skill was tried on #141 (fast-uri) before the fix-and-review loop was added. See the comment there: a build-only package, verified by the build and dev server, with @joaquimds requested as reviewer. The next daily run will put #141 through the review loop.

Before merging

🤖 Generated with Claude Code

https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1

The skill finds where a bumped package touches our code, adds the
smallest test of that integration, proves it red and green against both
our code and the library, and pushes it to the Dependabot branch for a
human to review. Once a package's tests are reviewed and recorded in
.github/dependency-coverage.json, later minor and patch bumps can
auto-merge on green CI. Major bumps always go to a human.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
Designers, engineers and headless Claude Code sessions all rely on the
join-flow dev server, and no CI job ran it. scripts/dev-server-smoke.sh
starts it and checks the page, the hot-reloading bundle and the stubbed
REST endpoints; the new Dev Server workflow runs it on every PR.

The verify-dependabot skill now requests review from whoever added the
dependency (or the top-level dependency that pulls in a transitive one),
falling back to whoever last changed it. scripts/bundled-packages.js
tells it whether a package ships in the bundle or is build tooling only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
Running it on #141 (fast-uri) showed:

- One ledger file meant every first-time PR edited the same JSON, so
  parallel Dependabot PRs would conflict. The ledger is now one file per
  package under .github/dependency-coverage/.
- Build-only packages still need the red/green proof, and the loader
  mutation has to hit a rule that runs (.scss; nothing imports .css).
- Updating a Dependabot branch from master should merge, not rebase and
  force-push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
@conatus conatus self-assigned this Oct 30, 2026
@conatus
conatus requested a review from joaquimds October 30, 2026 11:54
On GitHub Actions webpack colours its output, putting escape codes
between "compiled" and "successfully", so the check never matched and
the Dev Server job failed after its 30 second wait. With --no-color the
log is plain text everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
Previously a failing test was reported to the human reviewer as it was.
Now the skill fixes our code until CI is green (new and existing tests,
build, lint, E2E), then has a fresh review agent run the code-review
skill on our commits, fixes what it finds and asks a new agent to review
again. Only when CI is green and a review agent is satisfied does it ask
a person to review. After three rounds it stops and asks the reviewer for
help instead. A status comment on the PR records the round, so the daily
run picks up where it stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
The review agent now posts its findings as inline review comments, and
the skill answers each thread: fixed ones get the commit and are
resolved, declined ones get the reason and stay open for the person
reviewing. That leaves a visible record of what was found and done,
instead of a summary only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QZFw9N8p8vhvtvA3LDZG1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants