feat(rubric): add human-calibrated V5.3-r3 candidate

Author: yyuan308

.agents/skills/grade-homework/SKILL.md

@@ -249,12 +249,15 @@ unambiguous equivalent expressions satisfy an `equivalent_form_accepted`
 requirement. A required canonical simplification can be withheld only when the
 course rubric explicitly declares it required.
 
-When a rubric declares a local-error or carry-forward rule, preserve the
-downstream final-result criterion for the direct consequence of that one local
-error. Treat a wrong formula, wrong governing relation, or invalid method as a
-method criterion, not as a local arithmetic error. Ignore extra work that is
-irrelevant to every declared criterion; evaluate extra work only when it
-directly contradicts a declared criterion.
+When a final-result criterion is separately declared, withhold it when the
+reported final result is wrong, even when an earlier local error caused that
+result. The no-double-count rule protects only dependent process criteria; it
+does not preserve an independently allocated final-result criterion. A course
+rubric may define a narrow exception only by explicitly declaring a conditional
+carry-forward rule. Treat a wrong formula, wrong governing relation, or invalid
+method as a method criterion, not as a local arithmetic error. Ignore extra
+work that is irrelevant to every declared criterion; evaluate extra work only
+when it directly contradicts a declared criterion.
 
 Freeze the grading protocol before student grading starts:
 

.claude/skills/grade-homework/SKILL.md

@@ -249,12 +249,15 @@ unambiguous equivalent expressions satisfy an `equivalent_form_accepted`
 requirement. A required canonical simplification can be withheld only when the
 course rubric explicitly declares it required.
 
-When a rubric declares a local-error or carry-forward rule, preserve the
-downstream final-result criterion for the direct consequence of that one local
-error. Treat a wrong formula, wrong governing relation, or invalid method as a
-method criterion, not as a local arithmetic error. Ignore extra work that is
-irrelevant to every declared criterion; evaluate extra work only when it
-directly contradicts a declared criterion.
+When a final-result criterion is separately declared, withhold it when the
+reported final result is wrong, even when an earlier local error caused that
+result. The no-double-count rule protects only dependent process criteria; it
+does not preserve an independently allocated final-result criterion. A course
+rubric may define a narrow exception only by explicitly declaring a conditional
+carry-forward rule. Treat a wrong formula, wrong governing relation, or invalid
+method as a method criterion, not as a local arithmetic error. Ignore extra
+work that is irrelevant to every declared criterion; evaluate extra work only
+when it directly contradicts a declared criterion.
 
 Freeze the grading protocol before student grading starts:
 

benchmark/core/model_runner.py

@@ -11,7 +11,11 @@
 from .model_policy import bind_model_release_policy
 from .packets import directory_digest, validate_packet_output_contract
 from .run_metadata import validate_run_metadata
-from .rubrics import execution_criterion_ids, execution_criterion_points
+from .rubrics import (
+    execution_criterion_ids,
+    execution_criterion_points,
+    execution_scoring_gates,
+)
 from .schema import (
     CONFIDENCE_LEVELS,
     GRADING_OUTPUT_CONTRACT_DEDUCTION_TRACE_V1,
@@ -1036,6 +1040,7 @@ def _validate_grade_payload(
         for record in records:
             permitted_criteria = execution_criterion_ids(rubric, record.question_id)
             criterion_points = execution_criterion_points(rubric, record.question_id)
+            scoring_gates = execution_scoring_gates(rubric, record.question_id)
             if permitted_criteria is None:
                 continue
             traced_criteria: set[str] = set()
@@ -1056,6 +1061,28 @@ def _validate_grade_payload(
                     raise ValueError(
                         f"{record.question_id} deduction_trace points_deducted must equal the declared criterion points"
                     )
+            declared_gate_ids = set(scoring_gates or {})
+            used_gate_ids = declared_gate_ids & traced_criteria
+            if used_gate_ids:
+                if len(used_gate_ids) != 1 or len(traced_criteria) != 1:
+                    raise ValueError(
+                        f"{record.question_id} scoring-gate deduction must be the only deduction_trace entry"
+                    )
+                gate_id = next(iter(used_gate_ids))
+                gate = scoring_gates[gate_id]
+                if abs(float(record.score) - float(gate["score_cap"])) > 1e-9:
+                    raise ValueError(
+                        f"{record.question_id} scoring-gate deduction requires score_cap"
+                    )
+                gate_trace = next(
+                    trace
+                    for trace in record.deduction_trace
+                    if trace.rubric_criterion == gate_id
+                )
+                if gate_trace.deduction_type != gate["deduction_type"]:
+                    raise ValueError(
+                        f"{record.question_id} scoring-gate deduction_type must match the declared gate"
+                    )
     if "total" not in payload:
         raise ValueError("total is required")
     # Leaf score rows have already passed the course-specific coverage, range,

benchmark/core/rubrics.py

@@ -5,7 +5,7 @@
 import re
 from typing import Any
 
-from .schema import CourseSpec, QuestionSpec
+from .schema import CourseSpec, DEDUCTION_TYPES, QuestionSpec
 
 
 REQUIRED_LEVELS = (
@@ -193,6 +193,11 @@ def validate_execution_contract_rubric(
                 question, question_id, course_question, criterion_ids
             )
         )
+        findings.extend(
+            _validate_execution_scoring_gates(
+                question, question_id, course_question, criterion_ids
+            )
+        )
 
     for question_id in _duplicates(question_ids):
         findings.append(f"duplicate rubric question ID: {question_id}")
@@ -223,11 +228,20 @@ def execution_criterion_ids(
         criteria = question.get("criteria")
         if not isinstance(criteria, list):
             return None
-        return {
+        criterion_ids = {
             criterion["id"]
             for criterion in criteria
             if isinstance(criterion, dict) and isinstance(criterion.get("id"), str)
         }
+        scoring_gates = question.get("scoring_gates", [])
+        if not isinstance(scoring_gates, list):
+            return None
+        criterion_ids.update(
+            gate["id"]
+            for gate in scoring_gates
+            if isinstance(gate, dict) and isinstance(gate.get("id"), str)
+        )
+        return criterion_ids
     return None
 
 
@@ -247,13 +261,57 @@ def execution_criterion_points(
         criteria = question.get("criteria")
         if not isinstance(criteria, list):
             return None
-        return {
+        criterion_points = {
             criterion["id"]: float(criterion["points"])
             for criterion in criteria
             if isinstance(criterion, dict)
             and isinstance(criterion.get("id"), str)
             and _is_number(criterion.get("points"))
         }
+        scoring_gates = question.get("scoring_gates", [])
+        if not isinstance(scoring_gates, list):
+            return None
+        for gate in scoring_gates:
+            if (
+                isinstance(gate, dict)
+                and isinstance(gate.get("id"), str)
+                and _is_number(gate.get("score_cap"))
+                and _is_number(question.get("max_score"))
+            ):
+                criterion_points[gate["id"]] = float(question["max_score"]) - float(
+                    gate["score_cap"]
+                )
+        return criterion_points
+    return None
+
+
+def execution_scoring_gates(
+    rubric: dict[str, Any], question_id: str
+) -> dict[str, dict[str, Any]] | None:
+    """Return declared execution-contract score caps keyed by gate ID."""
+
+    if rubric.get("rubric_format") != EXECUTION_CONTRACT_FORMAT:
+        return None
+    questions = rubric.get("questions")
+    if not isinstance(questions, list):
+        return None
+    for question in questions:
+        if not isinstance(question, dict) or _question_id(question) != question_id:
+            continue
+        scoring_gates = question.get("scoring_gates", [])
+        if not isinstance(scoring_gates, list):
+            return None
+        return {
+            gate["id"]: {
+                "score_cap": float(gate["score_cap"]),
+                "deduction_type": gate["deduction_type"],
+            }
+            for gate in scoring_gates
+            if isinstance(gate, dict)
+            and isinstance(gate.get("id"), str)
+            and _is_number(gate.get("score_cap"))
+            and isinstance(gate.get("deduction_type"), str)
+        }
     return None
 
 
@@ -326,6 +384,49 @@ def _validate_execution_criteria(
     return findings
 
 
+def _validate_execution_scoring_gates(
+    question: dict[str, Any],
+    question_id: str,
+    course_question: QuestionSpec,
+    criterion_ids: list[str],
+) -> list[str]:
+    if "scoring_gates" not in question:
+        return []
+    scoring_gates = question["scoring_gates"]
+    if not isinstance(scoring_gates, list) or not scoring_gates:
+        return [f"{question_id} scoring_gates must be a non-empty list when present"]
+
+    findings: list[str] = []
+    required = {"id", "score_cap", "trigger", "deduction_type"}
+    for index, gate in enumerate(scoring_gates):
+        label = f"{question_id} scoring_gates[{index}]"
+        if not isinstance(gate, dict) or set(gate) != required:
+            findings.append(
+                f"{label} must contain exactly id, score_cap, trigger, deduction_type"
+            )
+            continue
+        gate_id = gate["id"]
+        if not isinstance(gate_id, str) or not gate_id.strip():
+            findings.append(f"{label} id must be non-blank text")
+        else:
+            criterion_ids.append(gate_id)
+        score_cap = gate["score_cap"]
+        if not _allows_score(course_question, score_cap) or _same_number(
+            score_cap, course_question.max_score
+        ):
+            findings.append(
+                f"{label} score_cap must be below the maximum and use the score step"
+            )
+        trigger = gate["trigger"]
+        if not isinstance(trigger, str) or not trigger.strip():
+            findings.append(f"{label} trigger must be non-blank text")
+        elif AMBIGUOUS_RUBRIC_LANGUAGE.search(trigger):
+            findings.append(f"{label} trigger must avoid unresolved discretionary language")
+        if gate["deduction_type"] not in DEDUCTION_TYPES:
+            findings.append(f"{label} deduction_type is invalid")
+    return findings
+
+
 def _validate_concept_question(
     rubric_question: dict[str, Any],
     question_id: str,

experiments/prompt_templates/grade_candidate_v5_3_r3.txt

@@ -0,0 +1,105 @@
+# Candidate Grading Prompt v5.3 r3
+
+You are grading anonymous student work in a reproducible blind experiment.
+
+Use only the files inside the current prompt packet:
+
+- `INSTRUCTIONS.md`
+- `manifest.json`
+- `course.json`
+- `rubric.json`
+- `output.schema.json`
+- files under `inputs/`, including each available `inputs/<student_id>/submission.json`
+
+For each anonymous student ID in `manifest.json`, grade only the supplied work
+against the frozen rubric. Produce exactly one JSON file per student under
+`outputs/`, named `<student_id>.json`. Do not inspect parent directories,
+other packets, prior outputs, identities, reference materials outside this
+packet, or other students' work.
+
+## Submission assembly
+
+The entire anonymous submission is the grading unit, never an individual page.
+Read every ordered page listed for the student in
+`inputs/<student_id>/submission.json`; that file is the authoritative page
+list. A page may contain evidence for multiple questions, and a question may
+continue across pages. Gather evidence for each declared leaf across the whole
+submission before assigning one score for that leaf.
+
+Page position, source-page number, image filename, and input index identify
+only source/display order. They are never question numbers or a question-to-
+page mapping. Do not assume that the first page is Q1 or that different
+submissions use the same physical question order. If the supplied content
+cannot reliably establish a relevant page, flag `page_order_uncertain` rather
+than assigning credit by position.
+
+## Scoring rules
+
+1. Score only the declared leaf items, their declared criteria, permitted score
+   increments, answer-only caps, and declared scoring gates. Do not merge,
+   borrow, average, curve, or offset credit across leaves.
+2. Record visible evidence before assigning points. Do not guess hidden intent,
+   repair an error, or infer unreadable work.
+3. Accept an unambiguous equivalent expression, notation variant, ordering, or
+   viable alternative method when it satisfies the same declared criterion. A
+   reference solution is an evidence anchor, not an exhaustive whitelist.
+4. Ignore extra work unless it directly contradicts a declared criterion. Do
+   not require a standard presentation when an equivalent valid route is
+   visible.
+5. For selected-response leaves, assess only the selected response unless the
+   stem explicitly requires explanation or working.
+6. For calculation leaves, distinguish a local notation/arithmetic error, a
+   wrong formula or governing relation, absent required work, failed required
+   simplification, and an incorrect final result. Preserve only independently
+   demonstrated process credit specified by the frozen rubric.
+7. A wrong final answer does not earn an independently allocated final-result
+   criterion merely because an earlier error caused it. The no-double-count rule
+   protects dependent process criteria, not a separately allocated final result.
+   Do not infer any carry-forward exception.
+8. When symbolic equivalence is required, check factored, expanded, reordered,
+   and otherwise unambiguous equivalent forms before withholding final-expression
+   credit. When the rubric makes a simplification conditional on the student's
+   preceding expression, evaluate that process criterion against that preceding
+   expression separately from the final-expression criterion.
+9. If a declared scoring-gate trigger is visibly present, apply its score cap
+   instead of the ordinary subtotal. Its deduction trace must contain only that
+   gate ID and its declared deduction type.
+10. Apply a correct answer with no required visible work only up to the frozen
+    answer-only cap.
+
+## Deduction trace contract
+
+For every non-full-credit leaf, emit a concise `deduction_trace` with one or
+more entries. Each entry has exactly:
+
+- `rubric_criterion`
+- `observed_evidence_or_missing_or_incorrect_part`
+- `deduction_type`
+- `points_deducted`
+
+Every trace must be checkable against visible work and the frozen rubric. Its
+point total must equal `max_score - score` for that leaf. Use the exact declared
+criterion or scoring-gate ID, and use its declared point value. This is a short
+audit statement, not hidden reasoning or a chain of thought.
+
+Full-credit leaves omit `deduction_trace`. A zero score requires a specific
+missing or incorrect reason. A leaf with flags or `low` confidence requires a
+brief `attention_note`. Treat bonus leaves as independent: never use a base-leaf
+deduction to explain, offset, or replace a bonus decision.
+
+Do not put a name, student identifier, email address, private path, raw file
+reference, or unneeded student answer transcription in a deduction trace or
+attention note.
+
+## Final checks
+
+Before writing output, revisit every unreadable or cropped region, blank or
+apparently missing answer, page-order concern, low-confidence score,
+high-impact deduction, total mismatch, and handwriting-dependent
+interpretation. Recheck semantic equivalence, score increments, arithmetic,
+alternative methods, scoring gates, and deduction-trace totals.
+
+Use exactly the JSON schema in `output.schema.json`. Recompute `total` from
+the leaf scores and apply the course total cap only after scoring every leaf,
+including any earned bonus leaf. Do not change the rubric, packet, score policy,
+or prompt while grading. Record uncertainty for course-owner review.