Pilot decision gate and annotation guidelines v0.2

README.md

@@ -154,7 +154,7 @@ item can be highlighted in the original advisory.
 | Phase | Scope | Status |
 |---|---|:---:|
 | 1 · Feasibility | Pinned ingestion, corpus statistics, label schema, offset-preserving preprocessing, pilot sample, annotation tooling | ✅ built |
-| 1 · Pilot | 100-document pilot imported (22 hand-labelled, 78 accepted model drafts); guideline revision, decision gate | 🟨 labels in · gate pending |
+| 1 · Pilot | 100-document pilot imported (22 hand-labelled, 78 accepted model drafts); [decision gate](docs/pilot_gate.md) measured; guidelines v0.2 drafted, frozen at `pilot-2` | 🟨 revision in progress |
 | 2 · Baselines | Rules baseline, CRF + logistic-regression attributes and groups, evaluator with bootstrap CIs, evidence viewer | ✅ built · preliminary pilot dev scores |
 | 3 · Neural models | Frozen leakage-safe splits, annotation batches, DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner | 🟨 code ready · training awaits labels |
 | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels |
@@ -251,13 +251,16 @@ JSON report. The same features are available as a JSON API: `POST /v1/extraction
 uv run triggermap corpus sample-pilot      # reproduce the pilot sample (seed 20261020)
 uv run triggermap annotate tasks           # Label Studio tasks for the sample
 uv run triggermap annotate import data/label_studio/export.json --annotator A1
+uv run triggermap annotate gate --export data/label_studio/export.json   # docs/pilot_gate.md
 ```
 
 The importer validates every exported annotation (offsets, labels, groups, document status),
 records its origin (manual, or a model draft accepted or edited), writes only clean records, and
 reports problems, label counts, origins and annotation pace. The procedure
 is in [`docs/pilot_protocol.md`](docs/pilot_protocol.md), and the labelling rules are in
-[`docs/annotation_guidelines.md`](docs/annotation_guidelines.md).
+[`docs/annotation_guidelines.md`](docs/annotation_guidelines.md). The gate projects each role
+and group operator to 800 documents from the non-test pilot records and marks classes below 50
+as unsupported.
 
 ## How results will be measured
 

annotations/README.md

@@ -28,7 +28,7 @@ Each record's `origin` states how its labels were made, read from Label Studio's
 
 | Origin | Meaning |
 |---|---|
-| `manual` | Labelled by hand, with no model draft. |
+| `manual` | Submitted without a Label Studio pre-annotation (labelled by hand in the tool). |
 | `model_draft_accepted` | A model pre-annotation submitted with its labels unchanged (`origin_model` names it). |
 | `model_draft_edited` | A model pre-annotation that a person changed before submitting. |
 

docs/annotation_guidelines.md

@@ -1,8 +1,10 @@
 # Annotation guidelines
 
-**Version 0.1 (pilot draft).** Label schema v1, preprocessing version 1.
-These rules are tested by the 100-document pilot and will change. Every change is recorded in
-the [changelog](#changelog), and annotations record the guideline version they followed.
+**Version 0.2 (post-pilot draft).** Label schema v1 (record schema 1.1), preprocessing version 1.
+Version 0.2 resolves the questions raised by the pilot (rules **R1–R8**, cite them in notes). It is
+frozen when the revised pilot is imported as `pilot-2` (section 13), and batch annotation follows
+it. Every change is recorded in the [changelog](#changelog), and annotations record the guideline
+version they followed.
 
 ## 1. Goal
 
@@ -42,6 +44,11 @@ matters for exposure?* If both answers are yes, it is a condition.
 | Workaround / mitigation | "disable the option immediately", "upgrade to 2.3.2" | Advice, even if it implies a condition. Annotate the condition only where the text states it as a condition. |
 | Generic best practice | "we recommend a reverse proxy" | Not stated as required for exposure. |
 | Severity, CVSS, credits, links | "CVSS 7.5", "thanks to …" | Not conditions. |
+| Payload construction (R1) | "a ZIP whose entry names do not match", "chains containing duplicate self-signed certificates" | How the attacker builds the input. A developer cannot check their deployment for it. |
+| Limits on impact (R6) | "access continues until the session expires" | Bounds how long or how far the impact goes, not whether the application is exposed. |
+| Behaviour after the fix | "the patched version now marks it unsafe" | Version matching covers fixed releases. |
+| Proof-of-concept specifics | "the PoC listens on port 8002" | Example values, unless the prose says the vulnerability requires them. |
+| Unspecified hedges (R4) | "in some circumstances", "under certain configurations" | Names nothing a developer could check. |
 
 ## 4. Categories
 
@@ -61,7 +68,22 @@ Tie-breakers, in order:
 3. A property of the attacker → `ACCESS`.
 4. Anything about the surrounding environment → `PLATFORM`.
 
-Write a note if none fits well. The pilot will tell us whether the categories need changing.
+Settled by the pilot:
+
+- **R1. Input.** *What kind of input the application processes* is `USAGE`: "processing
+  untrusted images", "parsing certificate chains supplied by clients". The span keeps the
+  adjectives inside that noun phrase (`untrusted`, `user-supplied`, `crafted`, `malicious`,
+  `invalid`) and stops before clauses that describe how the payload is built ("… containing
+  duplicate self-signed certificates" is left out). *The attacker's ability to deliver it* is
+  `ACCESS`: "attacker control of the chain input", "can upload files".
+- **R2. Request parameters.** An option the attacker sets in their own request
+  (`trust_remote_code=true` sent in the request) is attacker capability, `ACCESS`. A server-side
+  setting that must *allow* that option is `CONFIGURATION`.
+- **R3. Events in the environment.** Something that must happen in the environment (a server
+  restart, a scheduled job running, a package being reinstalled) is `PLATFORM`. Something the
+  victim must do stays `ACCESS`.
+
+Write a note if none fits well.
 
 ## 5. Roles
 
@@ -84,6 +106,29 @@ Write a note if none fits well. The pilot will tell us whether the categories ne
 Ask: *is the negation part of what must be true (include it, `REQUIRED`), or is it the author
 saying something is unnecessary (leave it out, `NOT_REQUIRED`)?*
 
+### Settled by the pilot
+
+- **R4. Hedging.** A hedge on *whether the vulnerability can be exploited* ("may allow", "can
+  lead to") does not change any role. A hedge on *whether a stated condition is needed*
+  ("mainly dangerous when X", "typically requires X", "in some configurations, such as X") makes
+  that condition `UNCERTAIN`, with a note. A hedge that names no checkable condition ("in some
+  circumstances") is not annotated.
+- **R5. Ineffective protections.** "Even with X enabled", "X does not prevent this",
+  "regardless of X": the text says the *absence* of X is not needed. Annotate `X` (without the
+  negating words) in its own category with role `NOT_REQUIRED`. Never `EXCEPTION`: X does not
+  protect. Example: "restrictive url_fetcher configuration does not prevent exploitation" →
+  `restrictive url_fetcher configuration`, CONFIGURATION, NOT_REQUIRED.
+- **R6. Exceptions remove exposure, not impact.** `EXCEPTION` is only for something that, if it
+  holds, means the application is not exposed at all. Something that only shortens or limits
+  the impact (session expiry, rate limits that slow an attack) is not a condition.
+- **R7. Variant- and escalation-specific conditions.** When the advisory describes several
+  attack variants, or a main vulnerability plus an escalated impact, a condition that the text
+  ties to **only one variant or only the escalation** is `UNCERTAIN`. Write a note naming the
+  variant ("variant: URL fetch to internal hosts"), and do not group it with conditions of the
+  main vulnerability. If the whole advisory is about that variant, its conditions are
+  `REQUIRED` as usual. This keeps conditions visible without presenting them as prerequisites
+  for every exposure.
+
 ## 6. Span boundaries
 
 A span is the **minimal contiguous phrase that fully states the condition**.
@@ -105,8 +150,20 @@ Splitting:
 - Two conditions coordinated with their own predicates ("using X **and** enabling Y") → two
   spans, plus a group (section 7).
 - An "or"/"and" **inside one description** ("a caching proxy that does not strip cookies
-  **or** ignore responses with cookies") → one span. Rule of thumb: separate spans when a
-  developer would check them separately.
+  **or** ignore responses with cookies", "kernels and terminals disabled or delegated to remote
+  hosts") → one span. Rule of thumb: separate spans when a developer would check them separately.
+
+Qualifiers that the text leaves unspecified stay as written. "An authenticated user with
+specific data permissions" is one `ACCESS` span. Never add the permission names, character
+sets or sizes the text does not give.
+
+References (**R8**):
+- When the only statement of a condition uses a reference ("the variable is not set", "that
+  page") and the referent is named **earlier in the same paragraph**, annotate the phrase as
+  written and name the referent in a note ("the variable = A2U_AUTH_TOKEN").
+- When the referent is outside the paragraph, annotate the phrase with role `UNCERTAIN`.
+- When the condition is also stated with its name elsewhere, annotate both statements (see
+  Repetition below).
 
 Code:
 - Annotate inside code only when the prose presents the code as the vulnerable setup
@@ -213,8 +270,38 @@ reported back for you to fix in Label Studio.
 Use the notes box whenever you hesitate: an unclear category, a borderline workaround, a span
 boundary you are unsure of. Notes are reviewed after the pilot to revise these guidelines.
 
+## 12. Unsupported classes (pilot gate)
+
+The pilot gate (`docs/pilot_gate.md`, regenerated with `triggermap annotate gate`) projects each
+role and group operator to the planned 800 documents. Classes projected below 50 examples are
+**unsupported**:
+
+| Class | Pilot gate result | What it means |
+|---|---|---|
+| `UNCERTAIN` | unsupported | Keep annotating as before. Models do not claim it; it is a *Review required* state. |
+| `ANY` | unsupported | Keep annotating. Predicted `ANY` groups are shown for review, and results are reported as low-support. |
+| `UNRESOLVED` | unsupported | Keep annotating; always *Review required*. |
+| `EXCEPTION` | supported in training only | No exceptions occurred in the natural pilot documents, so dev/test support will be very small. Training examples come from enriched batches. |
+
+Unsupported means *too few examples to learn or measure*, not *removed*. Never invent or
+adapt examples to fill a class.
+
+## 13. Migrating the pilot to v0.2
+
+`pilot-1` was labelled under v0.1 and stays as it is. Before batch-01:
+
+1. In Label Studio, re-check every pilot task that R1–R8 could affect: any note or draft review
+   flag about crafted input, ineffective protections, hedging, impact-specific or
+   variant-specific conditions, request parameters, lifecycle events or references. Edit,
+   resubmit, and cite the rule in the note ("R5").
+2. Export and import as a new version:
+   `triggermap annotate import data/label_studio/export.json --annotator A1 --annotation-version pilot-2 --out annotations/pilot/pilot-2.jsonl`.
+3. Regenerate the gate (`triggermap annotate gate --gold annotations/pilot/pilot-2.jsonl --export data/label_studio/export.json`).
+   Version 0.2 is frozen at that point.
+
 ## Changelog
 
 | Version | Date | Change |
 |---|---|---|
 | 0.1 | 2026-10-20 | Pilot draft. |
+| 0.2 | 2026-11-02 | Post-pilot draft. R1 input vs payload construction; R2 attacker-set request parameters are ACCESS; R3 environment events are PLATFORM; R4 hedging; R5 ineffective protections are NOT_REQUIRED; R6 exceptions remove exposure, not impact; R7 variant/escalation-specific conditions are UNCERTAIN; R8 references. New "not conditions" rows, unspecified qualifiers, unsupported classes from the pilot gate (UNCERTAIN, ANY, UNRESOLVED), pilot migration to `pilot-2`. |

docs/annotation_plan.md

@@ -6,7 +6,7 @@ in `splits/pool-v1.json` (see [`splits.md`](splits.md)). Every batch is committe
 
 | Sample | Pool | n | Strategy | Usable for | When to annotate |
 |---|---|---:|---|---|---|
-| `pilot/pilot_manifest.json` | mixed | 100 | 50 natural + 50 enriched | 39 train, 38 dev, 12 test (11 excluded) | now (guidelines v0.1) |
+| `pilot/pilot_manifest.json` | mixed | 100 | 50 natural + 50 enriched | 39 train, 38 dev, 12 test (11 excluded) | done under v0.1 (`pilot-1`); revised to v0.2 as `pilot-2` |
 | `batches/batch-01.json` | train | 100 | natural | train | after guidelines v0.2 |
 | `batches/batch-02.json` | train | 100 | enriched (cue top quartile, ≤ 800 words) | train | after guidelines v0.2 |
 | `batches/batch-03.json` | dev | 112 | natural, ≤ 3 per leakage group | dev | after guidelines v0.2 |

docs/pilot_gate.md

@@ -0,0 +1,28 @@
+# Pilot decision gate
+
+Generated by `triggermap annotate gate`; do not edit by hand.
+
+- Gold: `annotations/pilot/pilot-1.jsonl`, split `splits/pool-v1.json`
+- Documents counted (test split excluded): 88 (enriched 50, random 38)
+- Origin of counted records: manual 18, model_draft_accepted 70
+- Projection: 467 natural + 333 enriched-like documents; evaluation support assumes 300 natural dev + test documents
+- Threshold: a class projected below 50 examples is **unsupported**
+
+| Group | Label | Random | Enriched | Manual only | Projected (800) | Projected in dev + test | Status |
+|---|---|---:|---:|---:|---:|---:|---|
+| Role | REQUIRED | 110 | 247 | 53 | 2997 | 868 | supported |
+| Role | EXCEPTION | 0 | 15 | 3 | 100 | 0 | supported · no natural evaluation support |
+| Role | NOT_REQUIRED | 3 | 8 | 2 | 90 | 24 | supported |
+| Role | UNCERTAIN | 3 | 0 | 1 | 37 | 24 | **unsupported** |
+| Group operator | ALL | 13 | 36 | 10 | 400 | 103 | supported |
+| Group operator | ANY | 2 | 3 | 0 | 45 | 16 | **unsupported** |
+| Group operator | UNRESOLVED | 1 | 2 | 0 | 26 | 8 | **unsupported** |
+| Document status | HAS_CONDITIONS | 34 | 50 | 17 | 751 | 268 | reported, not gated |
+| Document status | NO_EXPLICIT_CONDITION | 4 | 0 | 1 | 49 | 32 | reported, not gated |
+| Document status | UNSUPPORTED_STRUCTURE | 0 | 0 | 0 | 0 | 0 | reported, not gated · no natural evaluation support |
+
+## Pace
+
+- Timed documents: 22 (Label Studio lead time; manual records only)
+- Median minutes per document: 5.6
+- Projected hours for 800 documents at the median: 74

docs/pilot_protocol.md

@@ -70,7 +70,14 @@ agreement.
 
 ## Decision gate
 
-These are proposed criteria for review together after the pilot:
+The measured result is in [`pilot_gate.md`](pilot_gate.md), generated from the non-test pilot
+records with:
+
+```bash
+uv run triggermap annotate gate --export data/label_studio/export.json
+```
+
+The criteria:
 
 - **Pace.** The measured median minutes per document sets the realistic size of the gold set
   within the 80–130 hour budget.