Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Contributing

- Open a GitHub Issue first for anything non-trivial: https://github.com/redevops-io/DataOpsBench/issues
- Pull requests against the default branch are welcome. Keep changes small and focused.
- New scenarios need a synthetic builder, hidden ground truth and a deterministic verifier (see README "Contributing").
- Run the checks before submitting: `python3 dataopsbench.py validate`
- Documentation changes: do not add numbers or performance/accuracy claims without a reproducible artifact in the repo, or label them
`Illustrative/synthetic`, `Internally reproduced`, `Externally reproduced` or `Field validated` as appropriate.
- Maintainer contact for anything else: redevops@redevops.io.
38 changes: 33 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

[![NVIDIA Inception](https://img.shields.io/badge/NVIDIA-Inception%20Program%20Member-76B900.svg)](https://www.nvidia.com/en-us/startups/)

A standardized, model- and framework-agnostic workload for **AI-assisted data operations** — the DataOps
analogue of TPC-C. It doesn't ask "can an LLM write SQL?" It asks:
**What it does.** A model- and framework-agnostic workload for **AI-assisted data operations**, *inspired by*
transaction-processing benchmarks such as TPC-C. It is **not** a standard, and it is not audited or governed by
any standards body. It doesn't ask "can an LLM write SQL?" It asks:

> Can an AI system **diagnose, repair, and verify a production data platform** — preserving correctness,
> lineage, evidence, and governance — when the relevant code and metadata no longer fit in one model
Expand All @@ -14,21 +15,46 @@ a direct model, LangGraph, OpenHands, Claude Code, Codex, Cursor, or your own ru
**identical** scenarios by the **same** deterministic gates. Two reference solvers are included so the
workload runs out of the box.

**Who it's for.** Researchers and engineers comparing how AI systems operate a data platform past the context
window; a mechanism test, not a leaderboard.

**Maturity: internally reproduced** (v0; no tags or releases). Only two result files are committed
(see Evidence); this is not externally reproduced.

**3-minute example** (no API keys, no model calls):
```bash
python3 dataopsbench.py validate # expect: RESULT 5/5 scenarios validate
```

**Evidence.** [`results/validate.txt`](results/validate.txt) (gate validation, 5/5) and
[`results/s21.txt`](results/s21.txt) (S21 sweep; single run, one 64K-context model). No other scenario has a
committed result.

**Limitations.** Synthetic data only; one model and one run for S21; model-run scripts need your own endpoint;
see [STATUS.md](STATUS.md). Contributing: [CONTRIBUTING.md](CONTRIBUTING.md). Security: [SECURITY.md](SECURITY.md).

Background: [why bounded integration becomes *required* past the context window](https://redevops.io/blog/does-formal-orchestration-beat-a-supervisor)
· [the DataOpsBench results write-up](https://redevops.io/blog/dataopsbench-operating-a-data-platform).

## Validate the gates (no API keys, no model calls)

```bash
python3 dataopsbench.py validate # RESULT 4/4 — each gate FAILS on the defect and PASSES on the fix
python3 dataopsbench.py validate # RESULT 5/5 — each gate FAILS on the defect and PASSES on the fix
python3 dataopsbench.py spec [S03] # scenario spec(s)
```

The `5/5` matches [`results/validate.txt`](results/validate.txt) ("RESULT 5/5 scenarios validate"). It covers
**five** scenarios (S03, S10, S14, S20, S21); S14L is listed below but is **not** part of `validate`. (An earlier
version of this README said 4/4.)

`validate` is the integrity check: it proves every scenario's gates **discriminate** — the seeded defect
fails them, the reference fix passes them all. A gate that passed on the defect would be worthless.

## Scenarios (v0)

Committed results: `results/validate.txt` (S03, S10, S14, S20, S21 gates) and `results/s21.txt` (S21 sweep).
**No committed results exist for S03, S10, S14, S14L or S20 beyond gate validation** (S14L has none at all).

| id | family | task | deterministic gates |
|----|--------|------|---------------------|
| **S03** | data quality | duplicate policy records after a retried load | no dup key · distinct count · premium total after dedup |
Expand Down Expand Up @@ -59,8 +85,10 @@ python3 run_judge.py 1 2 3 4

**Arm A** is a direct model over the raw platform. **Arm B** is a bounded reference (chunk under a token
budget; for consolidation a bounded model pass, for localization a deterministic scan). The observed
pattern: the direct arm succeeds while the platform fits the window, then overflows; a bounded solver stays
feasible at any scale.
pattern on S21 (*Internally reproduced*: a single run, one 64K-context model, see [`results/s21.txt`](results/s21.txt)):
the direct arm succeeded at scales 20 and 100 and overflowed at 300 and 600; the bounded merge stayed correct at
every tested scale, up to scale 600 in the arm sweep and scale 2000 in the Spark run (263 executor tasks). Behaviour
beyond the largest tested scale, and for other scenarios or models, is not shown.

## Plug in your own solver (the arm interface)

Expand Down
7 changes: 7 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Security policy

To report a vulnerability, please email redevops@redevops.io (the ReDevOps organisation contact) and avoid
posting exploit details in a public issue. A dedicated security contact, response times and a disclosure policy are
**to be confirmed**; none is promised here.

Maturity note: this repository is labelled **internally reproduced** (see [STATUS.md](STATUS.md)); it has had no formal security audit.
21 changes: 21 additions & 0 deletions STATUS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Status

- **Maturity:** internally reproduced
- **Version:** v0 scenarios; no tags, releases or package version
- **Last git activity on the default branch:** 2026-09-25 (from `git log -1 --format=%cs`; the working clone is shallow)

## What is supported
- python3 dataopsbench.py validate (deterministic, no API keys): RESULT 5/5 scenarios validate (results/validate.txt)
- S21 sweep results committed in results/s21.txt (single run, one 64K-context model)
- Reference solvers via run_arm.py / run_judge.py against your own OpenAI-compatible endpoint

## What is not supported / not claimed
- Not a standard or audited benchmark; no standards body, spec versioning or leaderboard
- No committed results for S03, S10, S14, S14L, S20 beyond gate validation
- No external reproduction; synthetic data only

## Known issues
- README previously said 4/4 while results/validate.txt says 5/5; fixed (S14L is not covered by validate)
- Statements about behaviour at larger scales than tested (s21: 600 arm sweep, 2000 Spark) are not evidenced

No quantitative claim in this repo has been externally reproduced or field validated.
Loading