Skip to content

Releases are published before the acceptance gate runs #343

Description

@leggetter

Problem

gh release create (and the GitHub UI equivalent) creates the tag and publishes the release in one step. The tag push is what triggers release.yml, whose first job is the acceptance suite. So the ordering is:

  1. GitHub Release published and publicly visible
  2. then acceptance runs
  3. then GoReleaser builds, then publish-npm

Measured on v2.5.0:

release published : 2026-08-13T12:07:08Z
release.yml started: 2026-08-13T12:07:10Z   # 2 seconds later

The acceptance job gates the artifacts (build-* are needs: [acceptance], publish-npm is needs: [build-*]) — it does not gate the announcement.

Failure mode: if acceptance fails, a published, non-draft vX.Y.Z release sits on the releases page with no binaries attached, nothing on npm/Homebrew/Scoop/Docker, and the tag already consumed. Recovering means deleting the release and tag, or burning a version.

Second gap: the pre-tag gate does not cover acceptance

.agents/skills/hookdeck-cli-release/SKILL.md requires statusCheckRollup on main to be SUCCESS before tagging. That check cannot include acceptance, because:

  • test.yml (what runs on main) deliberately excludes it — "compile/unit/build only — acceptance tests stay PR-gated because they hit live APIs and are slow/flaky"
  • test-acceptance.yml triggers on pull_request only

So main's rollup is unit-test + build-* + govulncheck. For v2.5.0 the code was covered, because acceptance passed on the PR head and the squash merge produced an identical tree — but that is a property of the merge, not of the gate. A main that received a direct push, or a squash that differs from the PR head, would tag with no acceptance signal at all.

Options

  1. Create the release as a draft, publish after the workflow succeeds. gh release create --draft, then a workflow step flips it to published once publish-npm completes. Keeps one tag, no visible half-released state.
  2. Run acceptance on main push and require it in the pre-tag rollup. Costs a live-API run per merge, which is what test.yml set out to avoid.
  3. Tag-only trigger, release created by the workflow. Push the bare tag; release.yml creates the GitHub Release itself after the gate. Notes would need to come from a file in the repo or a workflow input.
  4. Document the ordering in the skill as a known limitation and require the releaser to confirm acceptance passed on the merged commit's PR before tagging. Cheapest, but relies on discipline.

Option 1 looks like the best value: it is a small change to the skill plus one workflow step, and it closes the visible-but-empty-release window without adding live-API runs to every merge.

Context

Spotted by @leggetter immediately after the v2.5.0 release, which was unaffected — the run succeeded and all artifacts published. Filing so the next one is not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions