Repository navigation
feat(evaluation): add an fcc scorer for tolerance and set-F1 scoring from scenario_meta.json - #648
Merged
Merged
Conversation
…from scenario_meta.json FCC scenarios name their scorer in scenario_meta.json, and an unregistered scoring_method aborts the whole evaluation with "unknown scorer". This adds a deterministic fcc scorer with two match types: - numeric: the answer must be within numeric_tolerance of the gold number, percentage (trailing % stripped), or list of numbers (same length, element-wise). A first line holding only the value is read directly, so numbers in an optional free-text line after it are ignored; otherwise the whole answer is parsed, then the first line as a fallback. - set_f1: the answer must be a list. F1 over distinct items, order- and case-insensitive, with objects compared as whole objects, must reach f1_pass_threshold. Both comparisons are inclusive and are made on values rounded to 6 decimals, as the fmea scorer does. Invalid metadata or ground truth fails the scenario with a rationale instead of raising. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a deterministic
fccscorer (src/evaluation/scorers/fcc.py) and registers it next tofmea. A scenario selects it, and one of two match types, in itsscenario_meta.json:{"scoring_method": "fcc", "match": "numeric", "numeric_tolerance": 1} {"scoring_method": "fcc", "match": "set_f1", "f1_pass_threshold": 0.8}numeric: the gold is a number, a percentage (trailing%stripped), or a list of numbers. The answer passes ifround(|answer - gold|, 6) <= numeric_tolerance. For a list, the answer must be a list of the same length with every element within tolerance. A first line holding only the value is read directly, so numbers in an optional free-text line after it are ignored. Otherwise the whole answer goes throughparse_structured_answer, and then the first line is tried as a fallback. Booleans never count as numbers. Score is 1.0 or 0.0.set_f1: gold and answer must be lists, and an answer that isn't a list scores F1 = 0. Distinct items are compared without regard to order. Scalars are compared throughnormalize_value, so case and whitespace don't matter. Objects are compared as whole objects, so a rightwo_idwith a wrongfailure_codecounts as one missing item and one extra item. F1 is2·tp / (|gold| + |answer|). The answer passes ifround(F1, 6) >= f1_pass_threshold, and the score isround(F1, 6).Both comparisons are inclusive and use values rounded to 6 decimals, the same rule the
fmeascorer uses since #646. If the metadata is bad (unknownmatch, a tolerance that is missing, negative or boolean, or a threshold that is not a number in [0, 1]) or the ground truth can't be parsed, the scenario fails with a rationale. The scorer never raises, so one bad file can't abort a run.detailscarries the gold and parsed values, the deltas or precision/recall/F1, the threshold and its operator, and the missing and extra items.Scenarios that use it
numericnumeric_tolerance: 1numericnumeric_tolerance: 3.3set_f1f1_pass_threshold: 0.8Every
set_f1scenario uses 0.8, the same threshold as the FMEA scenarios. Thesescenario_meta.jsonfiles live in the scenario-generation repo and are not committed yet. Until this scorer is merged, those files would make the evaluator raiseunknown scorer 'fcc'.Tests
src/evaluation/tests/test_fcc_scorer.py, 44 cases using synthetic values. It covers inclusive tolerance edges (including float error at the edge), percentages, element-wise lists and wrong lengths, the first-line rule and its fallback, F1 exactly at the threshold, order, case and duplicates, non-list answers, whole-object pair matching, invalid metadata or ground truth without raising, and registry lookup.python -m pytest src/evaluation/tests: 172 passed.Integration check (not committed)
I loaded
scenarios_data/with the evaluator's loader. Exactly the 14 scenarios above resolve toscoring_method == "fcc", and each scenario's own ground truth passes as the answer (14/14). Below are pass counts on the 264 recorded runs for these scenarios,fccagainststatic_json. They are indicative only, because these runs predate a ground-truth correction.fccpassstatic_jsonpass