Skip to content

About

Measure data agents like you measure models — scenario question banks, entity-level P/R/F1, failure taxonomy, resumable runs

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

data-agent-eval

Measure data agents like you measure models — with a benchmark, not vibes.

A framework for evaluating natural-language data agents (NL→SQL, metric lookup, lineage tracing) against scenario-based question banks with entity-level precision/recall/F1 scoring, resumable runs, and a single-file HTML report you can actually send to management.

question bank ──> runner ──> agent answers ──> entity judge ──> P/R/F1 + HTML report
   (scenario × N)     (resumable, parallel)      (typed, not prose)

Why entity-level scoring

Data answers fail in specific ways: right table, wrong time grain; right metric, wrong brand; right trend, wrong sign. Sentence-similarity scoring buries these failures. data-agent-eval judges the entities the answer should contain — tables, metrics, dimensions, values — so a report tells you which failure mode dominates, not just "72% good".

Failure taxonomy built into the judge:

Mode Meaning
TERM_MISS the business term never resolved to the governed metric/table
SCOPE_ERROR right entity, wrong grain/filter (time, brand, region)
MISSING_EVIDENCE conclusion without the supporting query/lineage
FABRICATION entity or value that exists nowhere in the platform
REFUSED agent declined — scored separately, never as silent failure

Methodology

  1. Scenario × question matrix. Pick ~10 real scenarios (trend, attribution, comparison, top-N, anomaly, impact…), ~50 questions each with expected entity sets. Questions come from actual analyst utterances, not generated ones.
  2. Run resumable. Long runs crash; the runner checkpoints per-question, so a 500-question run is 500 restartable units.
  3. Judge entities, not prose. Each answer is parsed for expected entities; exact + normalized matching (aliases, case, units).
  4. Report per failure mode. The one-line conclusion is always: which failure mode costs you the most accuracy, and therefore which fix (term dictionary? scope guardrail? retrieval?) buys the most points.

See docs/methodology.md for the full design.

Repo layout

data-agent-eval/
├── docs/methodology.md   # eval design, scoring, taxonomy
├── runner/               # resumable execution engine (planned v0.1)
├── judge/                # entity extraction + matching (planned v0.1)
├── report/               # single-file HTML report template
└── bank/                 # example scenario question bank (synthetic, no real data)

Status

v0 — methodology published, runner in progress. The methodology is the reusable part today: you can apply the scenario matrix + entity judging to your own agent this week with a notebook.

Roadmap

  • v0.1 — runner + judge, single backend, CSV/HTML output
  • v0.2 — parallel runs, cost/latency tracking per question
  • v0.3 — regression mode (run on PR, fail the build on accuracy drop)
  • v1.0 — public leaderboard format for NL→SQL/data-agent systems

License

MIT

About

Measure data agents like you measure models — scenario question banks, entity-level P/R/F1, failure taxonomy, resumable runs

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors