From 773590ce97a7c124dad66b935e532583055a8924 Mon Sep 17 00:00:00 2001 From: Alex Mats Date: Thu, 15 Oct 2026 19:38:58 -0400 Subject: [PATCH 1/2] Docs: reconcile validate 5\/5, list committed results, soften TPC-C analogy and scale claim, add STATUS\/CONTRIBUTING\/SECURITY Co-Authored-By: Claude Sonnet 5.5 --- CONTRIBUTING.md | 9 +++++++++ README.md | 38 +++++++++++++++++++++++++++++++++----- SECURITY.md | 9 +++++++++ STATUS.md | 21 +++++++++++++++++++++ 4 files changed, 72 insertions(+), 5 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md create mode 100644 STATUS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..82075c1 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,9 @@ +# Contributing + +- Open a GitHub Issue first for anything non-trivial: https://github.com/redevops-io/DataOpsBench/issues +- Pull requests against the default branch are welcome. Keep changes small and focused. +- New scenarios need a synthetic builder, hidden ground truth and a deterministic verifier (see README "Contributing"). +- Run the checks before submitting: `python3 dataopsbench.py validate` +- Documentation changes: do not add numbers or performance/accuracy claims without a reproducible artifact in the repo, or label them + `Illustrative/synthetic`, `Internally reproduced`, `Externally reproduced` or `Field validated` as appropriate. +- Maintainer contact for anything else: redevops@redevops.io (listed in the GitHub organisation metadata; a dedicated contributor channel is to be confirmed). diff --git a/README.md b/README.md index 4fce5ed..a27cba8 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,9 @@ [![NVIDIA Inception](https://img.shields.io/badge/NVIDIA-Inception%20Program%20Member-76B900.svg)](https://www.nvidia.com/en-us/startups/) -A standardized, model- and framework-agnostic workload for **AI-assisted data operations** — the DataOps -analogue of TPC-C. It doesn't ask "can an LLM write SQL?" It asks: +**What it does.** A model- and framework-agnostic workload for **AI-assisted data operations**, *inspired by* +transaction-processing benchmarks such as TPC-C. It is **not** a standard, and it is not audited or governed by +any standards body. It doesn't ask "can an LLM write SQL?" It asks: > Can an AI system **diagnose, repair, and verify a production data platform** — preserving correctness, > lineage, evidence, and governance — when the relevant code and metadata no longer fit in one model @@ -14,21 +15,46 @@ a direct model, LangGraph, OpenHands, Claude Code, Codex, Cursor, or your own ru **identical** scenarios by the **same** deterministic gates. Two reference solvers are included so the workload runs out of the box. +**Who it's for.** Researchers and engineers comparing how AI systems operate a data platform past the context +window; a mechanism test, not a leaderboard. + +**Maturity: internally reproduced** (v0; no tags or releases). Only two result files are committed +(see Evidence); this is not externally reproduced. + +**3-minute example** (no API keys, no model calls): +```bash +python3 dataopsbench.py validate # expect: RESULT 5/5 scenarios validate +``` + +**Evidence.** [`results/validate.txt`](results/validate.txt) (gate validation, 5/5) and +[`results/s21.txt`](results/s21.txt) (S21 sweep; single run, one 64K-context model). No other scenario has a +committed result. + +**Limitations.** Synthetic data only; one model and one run for S21; model-run scripts need your own endpoint; +see [STATUS.md](STATUS.md). Contributing: [CONTRIBUTING.md](CONTRIBUTING.md). Security: [SECURITY.md](SECURITY.md). + Background: [why bounded integration becomes *required* past the context window](https://redevops.io/blog/does-formal-orchestration-beat-a-supervisor) · [the DataOpsBench results write-up](https://redevops.io/blog/dataopsbench-operating-a-data-platform). ## Validate the gates (no API keys, no model calls) ```bash -python3 dataopsbench.py validate # RESULT 4/4 — each gate FAILS on the defect and PASSES on the fix +python3 dataopsbench.py validate # RESULT 5/5 — each gate FAILS on the defect and PASSES on the fix python3 dataopsbench.py spec [S03] # scenario spec(s) ``` +The `5/5` matches [`results/validate.txt`](results/validate.txt) ("RESULT 5/5 scenarios validate"). It covers +**five** scenarios (S03, S10, S14, S20, S21); S14L is listed below but is **not** part of `validate`. (An earlier +version of this README said 4/4.) + `validate` is the integrity check: it proves every scenario's gates **discriminate** — the seeded defect fails them, the reference fix passes them all. A gate that passed on the defect would be worthless. ## Scenarios (v0) +Committed results: `results/validate.txt` (S03, S10, S14, S20, S21 gates) and `results/s21.txt` (S21 sweep). +**No committed results exist for S03, S10, S14, S14L or S20 beyond gate validation** (S14L has none at all). + | id | family | task | deterministic gates | |----|--------|------|---------------------| | **S03** | data quality | duplicate policy records after a retried load | no dup key · distinct count · premium total after dedup | @@ -59,8 +85,10 @@ python3 run_judge.py 1 2 3 4 **Arm A** is a direct model over the raw platform. **Arm B** is a bounded reference (chunk under a token budget; for consolidation a bounded model pass, for localization a deterministic scan). The observed -pattern: the direct arm succeeds while the platform fits the window, then overflows; a bounded solver stays -feasible at any scale. +pattern on S21 (*Internally reproduced*: a single run, one 64K-context model, see [`results/s21.txt`](results/s21.txt)): +the direct arm succeeded at scales 20 and 100 and overflowed at 300 and 600; the bounded merge stayed correct at +every tested scale, up to scale 600 in the arm sweep and scale 2000 in the Spark run (263 executor tasks). Behaviour +beyond the largest tested scale, and for other scenarios or models, is not shown. ## Plug in your own solver (the arm interface) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..fff517c --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,9 @@ +# Security policy + +GitHub private vulnerability reporting is **not enabled** for this repository (checked via the GitHub API on 2026-10-15). + +To report a vulnerability, please email redevops@redevops.io (the address listed in the GitHub organisation metadata) and avoid +posting exploit details in a public issue. A dedicated security contact, response times and a disclosure policy are +**to be confirmed**; none is promised here. + +Maturity note: this repository is labelled **internally reproduced** (see [STATUS.md](STATUS.md)); it has had no formal security audit. diff --git a/STATUS.md b/STATUS.md new file mode 100644 index 0000000..d4b0d72 --- /dev/null +++ b/STATUS.md @@ -0,0 +1,21 @@ +# Status + +- **Maturity:** internally reproduced +- **Version:** v0 scenarios; no tags, releases or package version +- **Last git activity on the default branch:** 2026-09-25 (from `git log -1 --format=%cs`; the working clone is shallow) + +## What is supported +- python3 dataopsbench.py validate (deterministic, no API keys): RESULT 5/5 scenarios validate (results/validate.txt) +- S21 sweep results committed in results/s21.txt (single run, one 64K-context model) +- Reference solvers via run_arm.py / run_judge.py against your own OpenAI-compatible endpoint + +## What is not supported / not claimed +- Not a standard or audited benchmark; no standards body, spec versioning or leaderboard +- No committed results for S03, S10, S14, S14L, S20 beyond gate validation +- No external reproduction; synthetic data only + +## Known issues +- README previously said 4/4 while results/validate.txt says 5/5; fixed (S14L is not covered by validate) +- Statements about behaviour at larger scales than tested (s21: 600 arm sweep, 2000 Spark) are not evidenced + +No quantitative claim in this repo has been externally reproduced or field validated. From 721bf4a446bbe855a803cedad932269210284334 Mon Sep 17 00:00:00 2001 From: Alex Mats Date: Thu, 15 Oct 2026 20:24:50 -0400 Subject: [PATCH 2/2] Docs: remove internal notes and speculative commentary Co-Authored-By: Claude Sonnet 5.5 --- CONTRIBUTING.md | 2 +- SECURITY.md | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 82075c1..40e266b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,4 +6,4 @@ - Run the checks before submitting: `python3 dataopsbench.py validate` - Documentation changes: do not add numbers or performance/accuracy claims without a reproducible artifact in the repo, or label them `Illustrative/synthetic`, `Internally reproduced`, `Externally reproduced` or `Field validated` as appropriate. -- Maintainer contact for anything else: redevops@redevops.io (listed in the GitHub organisation metadata; a dedicated contributor channel is to be confirmed). +- Maintainer contact for anything else: redevops@redevops.io. diff --git a/SECURITY.md b/SECURITY.md index fff517c..899325f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,8 +1,6 @@ # Security policy -GitHub private vulnerability reporting is **not enabled** for this repository (checked via the GitHub API on 2026-10-15). - -To report a vulnerability, please email redevops@redevops.io (the address listed in the GitHub organisation metadata) and avoid +To report a vulnerability, please email redevops@redevops.io (the ReDevOps organisation contact) and avoid posting exploit details in a public issue. A dedicated security contact, response times and a disclosure policy are **to be confirmed**; none is promised here.