Skip to content

About

Benchleaking Is Accelerating

Resources

Stars

21 stars

Watchers

0 watching

Forks

Latest commit

 

History

14 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Benchleaking Is Accelerating

From hidden-test recall in agents to test-set leakage in base models.

Read the Report · Key Findings · Anti-Hack Scores · Code · Evaluation Protocol

Benchmark contamination has escalated over the past month, from hidden-test recall in post-trained agents to test-set leakage in public base models. We document two warning signs: agents stating hidden grading rules absent from their instructions, and base models reproducing complete benchmark test examples absent from their input. Our base-model study spans 20 benchmarks and four public base models. We introduce Anti-Hack scores to quantify how much reported performance depends on leaked examples, revealing substantial score drops and ranking changes. Because open base models underpin downstream research, the consequences of benchleaking extend beyond leaderboard rankings.

Overview: ERNIE-5.5-Flash-Base reproduces consecutive MMLU test questions; zeroing leaked examples changes model rankings.

Key Findings

  • Signs of benchleaking in post-training. Our MaxxBench replication finds drops of 13.3 points for DeepSeek-V4.1-Flash and 15.8 points for Kimi-K3.1-Pro on modified tasks. Our Terminal-Bench probes find hidden-rule candidates in Kimi-K3.1-Pro on 14.3% of tasks during normal solving and 51.5% under prefilled reasoning.
  • Test-set contamination in pre-training. ERNIE-5.5-Flash-Base reproduces 96.3% of HumanEval and 85.4% of OpenBookQA test examples absent from its input, including reference implementations or correct answers. Continuations can follow the original test-set order and preserve source typos such as examble.
  • Anti-Hack scores change model rankings. Giving detected leaked examples zero credit lowers ERNIE-5.5-Flash-Base's average by 16.6 points, moving it from first to last among the four models. MiMo-V2.5-Base loses 5.0 points, while DeepSeek-V4-Flash-Base and Hy3-preview-Base lose 1.5 and 1.3 points, respectively.

We distinguish benchmaxxing, optimization toward a benchmark's distribution, formats, or graders, from benchleaking, exposure to the benchmark's test content itself. Poor generalization alone does not establish test-set leakage.

What Leakage Looks Like

Given HumanEval/66, ERNIE-5.5-Flash-Base answers the question and continues with HumanEval/67 through HumanEval/70. Those four problems are absent from the input. The figure shows fruit_distribution in full, including its task description, reference implementation, and original typo.

HumanEval example: the input asks HumanEval/66, while the continuation reproduces subsequent test problems and preserves the typo examble.

Selected Leakage Rates

Percentage of distinct test examples reproduced under our strict matching criterion:

Base model HumanEval OpenBookQA GSM8K GPQA-Main
DeepSeek-V4-Flash-Base 9.1% 0.0% 0.8% 0.0%
Hy3-preview-Base 6.7% 0.0% 5.1% 0.0%
MiMo-V2.5-Base 73.8% 0.0% 4.4% 0.0%
ERNIE-5.5-Flash-Base 96.3% 85.4% 80.1% 34.6%

See Table 4 in the report for all 20 benchmarks.

Anti-Hack Scores

Anti-Hack marks detected leaked examples as incorrect while keeping the test-set denominator unchanged. The following scores are averages over 20 benchmark entries; drops are in percentage points.

Base model Original Anti-Hack Drop
ERNIE-5.5-Flash-Base 85.17 68.55 16.6
MiMo-V2.5-Base 80.01 75.03 5.0
DeepSeek-V4-Flash-Base 77.95 76.48 1.5
Hy3-preview-Base 79.86 78.55 1.3

Original and Anti-Hack scores on HumanEval, OpenBookQA, GSM8K, GPQA-Main, MMLU, and MBPP for four base models.

This adjustment shows how much reported performance depends on detected leaked examples.

Agent Audits

We replicate two complementary studies:

  • MaxxBench: on 89 shared tasks, DeepSeek-V4.1-Flash falls from 75.8% to 62.5%, and Kimi-K3.1-Pro from 83.8% to 68.0%, when task requirements or starting code are modified.
  • R40's Kimi K3.1 audit: we probe whether agents state hidden grading rules absent from the instructions. The table below reports our replication, not the original report's 23.3% figure.
Hidden-rule probe Kimi-K3.1-Pro Kimi-K3
Normal solving, 63 tasks 14.3% (9/63) 1.6% (1/63)
Prefilled reasoning, 66 tasks 51.5% (34/66) 9.1% (6/66)

These are task-level candidate rates from an automatic semantic judge, with one attempt per task. They are signs of hidden-test recall, distinct from the complete textual matches in the base-model study.

Evaluation Protocol

  1. Generate continuations. Use benchmark-specific few-shot prompts and allow base models to continue beyond the current answer.
  2. Match against the test set. Require the complete question, all required options or context, and the correct reference answer to occur together in a generated item. Exclude the current question and few-shot examples already in the input. Normalize case and whitespace; do not accept paraphrases or merely equivalent answers.
  3. Count distinct examples. Each matched test example counts once, even if generated repeatedly. For code tasks, require the reference implementation rather than an alternative passing solution.
  4. Adjust standard evaluation scores. Use the detected examples to compute Anti-Hack scores from separate standard evaluation runs.

The code directory contains leakage extraction, L1-L3 matching, and Anti-Hack score recomputation for the 20 base-model benchmarks, with runnable synthetic examples and tests. Matching rules, additional examples, full result tables, and evaluation configurations are documented in Appendices A-F of the report.

Public base models cannot be assumed to provide a clean starting point for downstream research. We call for routine decontamination reporting and contamination-adjusted evaluation.

Citation

@misc{qin2026benchleaking,
  title = {Benchleaking Is Accelerating},
  author = {Qin, Yiwei and Xie, Muhang and Li, Xuefeng
            and Jiang, Mohan and Fu, Dayuan and Mi, Tiantian
            and Guo, Qipeng and Liu, Pengfei},
  year = {2026},
  howpublished = {Technical report},
  url = {https://github.com/GAIR-NLP/Benchleaking}
}

About

Benchleaking Is Accelerating

Resources

Stars

21 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages