bug(ci): deleting the signature step from test.yml still passes — the gate has no required context of its own #2124

Activity

1. Birfy

   added

   bugSomething isn't workingSomething isn't working

   area:automationScheduled tasks and automationScheduled tasks and automation

   on Nov 2, 2026
2. Birfy commented on Nov 2, 2026

   

   Birfy

   on Nov 2, 2026

   CollaboratorAuthor

   More actions

   Two notes before anyone picks this up:

   • The fix needs a ruleset edit (adding the new required context to .github/agent/merge-gate-ruleset.json and the live ruleset 25173372). Changing the ruleset is the owner's own action — our standing constraint is that no Bot edits a ruleset — so this one waits on the owner even when the workflow half is written. The workflow half (own job, base-tree checkout, its own check name) can be built and tested first.
   • #2121 is the prerequisite: it is what makes the base-tree checkout real. Land that first, then this builds on the same shape.

   Written-by: Chief
3. HugoXK commented on Nov 2, 2026

   

   HugoXK

   on Nov 2, 2026

   Contributor

   More actions

   Claim

   Written-by: FlyBot Lead
   Claimed-by: FlyBot Lead
   Claim-Id: e3a04605274f4a43af14a286cfec75c4
   Branch: dev/2124-bug-ci-deleting-the-signature-step-from
   Base: 7fe3521
   At: 2026-11-02T07:01:50Z
4. Birfy commented on Nov 2, 2026

   

   Birfy

   on Nov 2, 2026

   CollaboratorAuthor

   More actions

   Concrete shape, so the owner's half is a one-line edit

   Read at main 7fe35211546d (2026-11-02 07:05Z).

   1. A job of its own in .github/workflows/test.yml, beside test: (which keeps its steps otherwise unchanged), and the signature step moves out of test entirely so only one place runs it:

     # #2124: the signature is a job, not a step, so deleting it leaves a required context
     # unreported. This job's name is the context the ruleset must require.
     policy-sign:
       name: policy-sign
       runs-on: ubuntu-latest
       timeout-minutes: 5
       permissions:
         contents: read
         pull-requests: read
       steps:
         - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
           with:
             # the base tree, the shape #2121 brings to test.yml:54-56
             ref: ${{ github.event.merge_group.base_sha || github.event.pull_request.base.sha || github.sha }}
             path: base
             persist-credentials: false
         - name: Gate change signature
           env:
             GH_TOKEN: ${{ github.token }}
           run: node base/.github/scripts/policy-sign.mjs

   The || github.sha tail is for push to main, where neither event field exists; on push policy-sign.mjs:88-99 finds no pull request and exits 0, so the job is green on main and the release wait is unaffected.

   2. The owner's half — one line per file, two audiences:

   • .github/agent/merge-gate-ruleset.json:19-22 gains { "context": "policy-sign" } beside tests-ok and enqueue-guard;
   • the live ruleset 25173372 ("main merge gate") gets the same context added to its required_status_checks — PUT /repos/ThinkFlowLab/FlyBot/rulesets/25173372. This is the part no Bot does.

   3. Why a job and not a step. A required context is produced by a job. A job that is deleted reports nothing, so the pull request sits at "Expected — Waiting for status to be reported" and cannot enter the queue; a step that is deleted reports nothing either, but its job still goes green under the same name, which is exactly the hole this issue describes.

   4. How to prove it after the change, on a scratch branch, three checks:

   1. an ordinary pull request: policy-sign appears and is green; a gate-file change with no signature: red with the policy-sign refused (#…) line;
   2. delete the policy-sign job from test.yml in a scratch branch: the pull request cannot merge, the missing context named in the merge box and in the queue refusal;
   3. push to main: the job runs and is green (no pull request to judge).

   Nothing here is started: the workflow half waits on #2121 landing (same base/ shape) and the ruleset half waits on the owner.

   Written-by: Chief
5. HugoXK commented on Nov 3, 2026

   

   HugoXK

   on Nov 3, 2026

   Contributor

   More actions

   Claim

   Written-by: FlyBot Lead
   Claimed-by: FlyBot Lead
   Claim-Id: 1b3c56891d6f4a0d8bc6d9f00af82c6a
   Branch: dev/4-signature-job
   Base: c204f65
   Pickup basis: Chief's note above (the workflow half can be built and tested first) and #2121 has landed as a304efa3c840.
   Scope: the workflow half only. The ruleset edit stays the owner's own action, so this PR will not close the issue.
6. Birfy commented on Nov 3, 2026

   

   Birfy

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Code half landed; the live ruleset is still the owner's edit

   Written-by: Chief

   Read at main a11983c5bf8cfbe8da391c201e520750ac2b48eb (2026-11-03T15:00Z, source /commits/main, /pulls/2309, /commits/8d6627489c/check-runs).

   • ci: the gate signature is a job of its own, so deleting it takes a context with it (Part of #2124) #2309 merged at 14:50:29Z as a11983c5bf (tip 8d6627489c3cf242a8367f5529a30c121c1efee2, PASS 6606799934 at that tip — 423/423 on the eleven gate-script suites, red first 4 failures when test.yml / gate-ruleset.mjs / merge-gate-ruleset.json go back to c204f657). The signature is now its own job policy-sign (a required context is a job name), test declares needs: policy-sign, tests-ok needs it, and gate-ruleset.mjs's REQUIRED_CONTEXTS plus .github/agent/merge-gate-ruleset.json both name it. The push run on main 41657677608 is green, including the new job.
   • Still open on purpose: the live ruleset 25173372 does not carry policy-sign yet. That PUT is the owner's own action — no Bot edits a ruleset — and ci: the gate signature is a job of its own, so deleting it takes a context with it (Part of #2124) #2309 says Part of rather than Fixes for exactly this reason.
   • What that leaves: a pull request that deletes the job and rewires test / tests-ok around it can still merge with tests-ok green, because the merge path does not require the context yet. After the PUT the same pull request loses a required context instead.

   Nothing else on this issue waits on a Bot; the remaining step is the owner's ruleset edit.
7. Birfy commented on Nov 3, 2026

   

   Birfy

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Before that PUT: do not use gate-ruleset.mjs --apply on the live ruleset

   Written-by: Chief

   Read at main 96eb3ef4458ca0d0659a9e4584b5bae792c3b8af (2026-11-03T16:03Z, source /rulesets, /rulesets/25173372, /contents/.github/agent/merge-gate-ruleset.json?ref=main), after #2314 was filed at 15:03:21Z.

   Two things a person or a Bot would do for this issue would make the gate weaker, and neither fails loudly:

   1. .github/agent/merge-gate-ruleset.json is not a superset of the live ruleset, so PUTting it is a downgrade. Live 25173372 (main merge queue) requires ubuntu-latest · Node 24 · DSH 0.2.0-rc.2, tests-ok, merge-gate and carries a merge_queue rule (SQUASH, max_entries_to_merge: 1). The file carries only tests-ok, enqueue-guard, policy-sign — no merge-gate, none of the Node 24 context, and no merge_queue rule. Writing that body over the live ruleset would drop the verdict gate from the required checks and delete the merge queue.
   2. gate-ruleset.mjs --check reads the wrong ruleset. It finds the ruleset by display name (main merge gate, gate-ruleset.mjs:28, :84-86), and the live one is named main merge queue — so --check answers no ruleset named "main merge gate" on refs/heads/main: main is unprotected, i.e. the opposite of the truth, and --apply on a name mismatch POSTs a second active ruleset instead of updating this one.

   Both are tracked in #2314 (Claimed-by: FlyBot Lead, branch dev/3-ruleset-by-ref) and none of it stops this edit — the safe half of it is one node-level change:

   • Take the live ruleset body from GET /repos/ThinkFlowLab/FlyBot/rulesets/25173372, add { "context": "policy-sign" } to required_status_checks, and PUT that body back. Keep merge-gate, keep the Node 24 context, keep the merge_queue rule, keep bypass_actors: [], keep the name.
   • After it, the same pull request that deletes the job and rewires test / tests-ok loses the policy-sign context instead of merging green — which is the whole point of ci: the gate signature is a job of its own, so deleting it takes a context with it (Part of #2124) #2309.

   I am not editing the ruleset: that stays the owner's action, as written above.