Measure data agents like you measure models — with a benchmark, not vibes.
A framework for evaluating natural-language data agents (NL→SQL, metric lookup, lineage tracing) against scenario-based question banks with entity-level precision/recall/F1 scoring, resumable runs, and a single-file HTML report you can actually send to management.
question bank ──> runner ──> agent answers ──> entity judge ──> P/R/F1 + HTML report
(scenario × N) (resumable, parallel) (typed, not prose)
Data answers fail in specific ways: right table, wrong time grain; right metric, wrong brand; right trend, wrong sign. Sentence-similarity scoring buries these failures. data-agent-eval judges the entities the answer should contain — tables, metrics, dimensions, values — so a report tells you which failure mode dominates, not just "72% good".
Failure taxonomy built into the judge:
| Mode | Meaning |
|---|---|
TERM_MISS |
the business term never resolved to the governed metric/table |
SCOPE_ERROR |
right entity, wrong grain/filter (time, brand, region) |
MISSING_EVIDENCE |
conclusion without the supporting query/lineage |
FABRICATION |
entity or value that exists nowhere in the platform |
REFUSED |
agent declined — scored separately, never as silent failure |
- Scenario × question matrix. Pick ~10 real scenarios (trend, attribution, comparison, top-N, anomaly, impact…), ~50 questions each with expected entity sets. Questions come from actual analyst utterances, not generated ones.
- Run resumable. Long runs crash; the runner checkpoints per-question, so a 500-question run is 500 restartable units.
- Judge entities, not prose. Each answer is parsed for expected entities; exact + normalized matching (aliases, case, units).
- Report per failure mode. The one-line conclusion is always: which failure mode costs you the most accuracy, and therefore which fix (term dictionary? scope guardrail? retrieval?) buys the most points.
See docs/methodology.md for the full design.
data-agent-eval/
├── docs/methodology.md # eval design, scoring, taxonomy
├── runner/ # resumable execution engine (planned v0.1)
├── judge/ # entity extraction + matching (planned v0.1)
├── report/ # single-file HTML report template
└── bank/ # example scenario question bank (synthetic, no real data)
v0 — methodology published, runner in progress. The methodology is the reusable part today: you can apply the scenario matrix + entity judging to your own agent this week with a notebook.
- v0.1 — runner + judge, single backend, CSV/HTML output
- v0.2 — parallel runs, cost/latency tracking per question
- v0.3 — regression mode (run on PR, fail the build on accuracy drop)
- v1.0 — public leaderboard format for NL→SQL/data-agent systems