From hidden-test recall in agents to test-set leakage in base models.
Read the Report · Key Findings · Anti-Hack Scores · Code · Evaluation Protocol
Benchmark contamination has escalated over the past month, from hidden-test recall in post-trained agents to test-set leakage in public base models. We document two warning signs: agents stating hidden grading rules absent from their instructions, and base models reproducing complete benchmark test examples absent from their input. Our base-model study spans 20 benchmarks and four public base models. We introduce Anti-Hack scores to quantify how much reported performance depends on leaked examples, revealing substantial score drops and ranking changes. Because open base models underpin downstream research, the consequences of benchleaking extend beyond leaderboard rankings.
- Signs of benchleaking in post-training. Our MaxxBench replication finds drops of 13.3 points for DeepSeek-V4.1-Flash and 15.8 points for Kimi-K3.1-Pro on modified tasks. Our Terminal-Bench probes find hidden-rule candidates in Kimi-K3.1-Pro on 14.3% of tasks during normal solving and 51.5% under prefilled reasoning.
- Test-set contamination in pre-training. ERNIE-5.5-Flash-Base reproduces 96.3% of HumanEval and 85.4% of OpenBookQA test examples absent from its input, including reference implementations or correct answers. Continuations can follow the original test-set order and preserve source typos such as
examble. - Anti-Hack scores change model rankings. Giving detected leaked examples zero credit lowers ERNIE-5.5-Flash-Base's average by 16.6 points, moving it from first to last among the four models. MiMo-V2.5-Base loses 5.0 points, while DeepSeek-V4-Flash-Base and Hy3-preview-Base lose 1.5 and 1.3 points, respectively.
We distinguish benchmaxxing, optimization toward a benchmark's distribution, formats, or graders, from benchleaking, exposure to the benchmark's test content itself. Poor generalization alone does not establish test-set leakage.
Given HumanEval/66, ERNIE-5.5-Flash-Base answers the question and continues with HumanEval/67 through HumanEval/70. Those four problems are absent from the input. The figure shows fruit_distribution in full, including its task description, reference implementation, and original typo.
Percentage of distinct test examples reproduced under our strict matching criterion:
| Base model | HumanEval | OpenBookQA | GSM8K | GPQA-Main |
|---|---|---|---|---|
| DeepSeek-V4-Flash-Base | 9.1% | 0.0% | 0.8% | 0.0% |
| Hy3-preview-Base | 6.7% | 0.0% | 5.1% | 0.0% |
| MiMo-V2.5-Base | 73.8% | 0.0% | 4.4% | 0.0% |
| ERNIE-5.5-Flash-Base | 96.3% | 85.4% | 80.1% | 34.6% |
See Table 4 in the report for all 20 benchmarks.
Anti-Hack marks detected leaked examples as incorrect while keeping the test-set denominator unchanged. The following scores are averages over 20 benchmark entries; drops are in percentage points.
| Base model | Original | Anti-Hack | Drop |
|---|---|---|---|
| ERNIE-5.5-Flash-Base | 85.17 | 68.55 | 16.6 |
| MiMo-V2.5-Base | 80.01 | 75.03 | 5.0 |
| DeepSeek-V4-Flash-Base | 77.95 | 76.48 | 1.5 |
| Hy3-preview-Base | 79.86 | 78.55 | 1.3 |
This adjustment shows how much reported performance depends on detected leaked examples.
We replicate two complementary studies:
- MaxxBench: on 89 shared tasks, DeepSeek-V4.1-Flash falls from 75.8% to 62.5%, and Kimi-K3.1-Pro from 83.8% to 68.0%, when task requirements or starting code are modified.
- R40's Kimi K3.1 audit: we probe whether agents state hidden grading rules absent from the instructions. The table below reports our replication, not the original report's 23.3% figure.
| Hidden-rule probe | Kimi-K3.1-Pro | Kimi-K3 |
|---|---|---|
| Normal solving, 63 tasks | 14.3% (9/63) | 1.6% (1/63) |
| Prefilled reasoning, 66 tasks | 51.5% (34/66) | 9.1% (6/66) |
These are task-level candidate rates from an automatic semantic judge, with one attempt per task. They are signs of hidden-test recall, distinct from the complete textual matches in the base-model study.
- Generate continuations. Use benchmark-specific few-shot prompts and allow base models to continue beyond the current answer.
- Match against the test set. Require the complete question, all required options or context, and the correct reference answer to occur together in a generated item. Exclude the current question and few-shot examples already in the input. Normalize case and whitespace; do not accept paraphrases or merely equivalent answers.
- Count distinct examples. Each matched test example counts once, even if generated repeatedly. For code tasks, require the reference implementation rather than an alternative passing solution.
- Adjust standard evaluation scores. Use the detected examples to compute Anti-Hack scores from separate standard evaluation runs.
The code directory contains leakage extraction, L1-L3 matching, and Anti-Hack score recomputation for the 20 base-model benchmarks, with runnable synthetic examples and tests. Matching rules, additional examples, full result tables, and evaluation configurations are documented in Appendices A-F of the report.
Public base models cannot be assumed to provide a clean starting point for downstream research. We call for routine decontamination reporting and contamination-adjusted evaluation.
@misc{qin2026benchleaking,
title = {Benchleaking Is Accelerating},
author = {Qin, Yiwei and Xie, Muhang and Li, Xuefeng
and Jiang, Mohan and Fu, Dayuan and Mi, Tiantian
and Guo, Qipeng and Liu, Pengfei},
year = {2026},
howpublished = {Technical report},
url = {https://github.com/GAIR-NLP/Benchleaking}
}

