Skip to content

feat(evaluation): add an fcc scorer for tolerance and set-F1 scoring from scenario_meta.json - #648

Merged
ShuxinLin merged 1 commit into
aafeedback_changesfrom
feat/fcc-scorer
Oct 19, 2026
Merged

ShuxinLin merged 1 commit into
aafeedback_changesfrom
feat/fcc-scorer

Conversation

@ShuxinLin

@ShuxinLin ShuxinLin commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds a deterministic fcc scorer (src/evaluation/scorers/fcc.py) and registers it next to fmea. A scenario selects it, and one of two match types, in its scenario_meta.json:

{"scoring_method": "fcc", "match": "numeric", "numeric_tolerance": 1}
{"scoring_method": "fcc", "match": "set_f1", "f1_pass_threshold": 0.8}
  • numeric: the gold is a number, a percentage (trailing % stripped), or a list of numbers. The answer passes if round(|answer - gold|, 6) <= numeric_tolerance. For a list, the answer must be a list of the same length with every element within tolerance. A first line holding only the value is read directly, so numbers in an optional free-text line after it are ignored. Otherwise the whole answer goes through parse_structured_answer, and then the first line is tried as a fallback. Booleans never count as numbers. Score is 1.0 or 0.0.
  • set_f1: gold and answer must be lists, and an answer that isn't a list scores F1 = 0. Distinct items are compared without regard to order. Scalars are compared through normalize_value, so case and whitespace don't matter. Objects are compared as whole objects, so a right wo_id with a wrong failure_code counts as one missing item and one extra item. F1 is 2·tp / (|gold| + |answer|). The answer passes if round(F1, 6) >= f1_pass_threshold, and the score is round(F1, 6).

Both comparisons are inclusive and use values rounded to 6 decimals, the same rule the fmea scorer uses since #646. If the metadata is bad (unknown match, a tolerance that is missing, negative or boolean, or a threshold that is not a number in [0, 1]) or the ground truth can't be parsed, the scenario fails with a rationale. The scorer never raises, so one bad file can't abort a run. details carries the gold and parsed values, the deltas or precision/recall/F1, the threshold and its operator, and the missing and extra items.

Scenarios that use it

match setting scenarios
numeric numeric_tolerance: 1 303, 306, 307, 308, 309, 310, 311, 318
numeric numeric_tolerance: 3.3 319, 323
set_f1 f1_pass_threshold: 0.8 304, 312, 316, 327

Every set_f1 scenario uses 0.8, the same threshold as the FMEA scenarios. These scenario_meta.json files live in the scenario-generation repo and are not committed yet. Until this scorer is merged, those files would make the evaluator raise unknown scorer 'fcc'.

Tests

  • New src/evaluation/tests/test_fcc_scorer.py, 44 cases using synthetic values. It covers inclusive tolerance edges (including float error at the edge), percentages, element-wise lists and wrong lengths, the first-line rule and its fallback, F1 exactly at the threshold, order, case and duplicates, non-list answers, whole-object pair matching, invalid metadata or ground truth without raising, and registry lookup.
  • python -m pytest src/evaluation/tests: 172 passed.

Integration check (not committed)

I loaded scenarios_data/ with the evaluator's loader. Exactly the 14 scenarios above resolve to scoring_method == "fcc", and each scenario's own ground truth passes as the answer (14/14). Below are pass counts on the 264 recorded runs for these scenarios, fcc against static_json. They are indicative only, because these runs predate a ground-truth correction.

scenario match runs fcc pass static_json pass
303 numeric 75 53 20
304 set_f1 9 7 0
306 numeric 10 4 0
307 numeric 10 8 7
308 numeric 10 4 3
309 numeric 10 5 1
310 numeric 8 0 0
311 numeric 10 7 1
312 set_f1 10 4 1
316 set_f1 10 7 0
318 numeric 74 45 16
319 numeric 10 8 1
323 numeric 9 6 0
327 set_f1 9 8 2
total 264 166 52

…from scenario_meta.json

FCC scenarios name their scorer in scenario_meta.json, and an unregistered
scoring_method aborts the whole evaluation with "unknown scorer". This adds
a deterministic fcc scorer with two match types:

- numeric: the answer must be within numeric_tolerance of the gold number,
  percentage (trailing % stripped), or list of numbers (same length,
  element-wise). A first line holding only the value is read directly, so
  numbers in an optional free-text line after it are ignored; otherwise the
  whole answer is parsed, then the first line as a fallback.
- set_f1: the answer must be a list. F1 over distinct items, order- and
  case-insensitive, with objects compared as whole objects, must reach
  f1_pass_threshold.

Both comparisons are inclusive and are made on values rounded to 6
decimals, as the fmea scorer does. Invalid metadata or ground truth fails
the scenario with a rationale instead of raising.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin
ShuxinLin merged commit 31ee5a7 into aafeedback_changes Oct 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant