Skip to content

About

No description, website, or topics provided.

Resources

Stars

78 stars

Watchers

0 watching

Forks

Latest commit

 

History

10 Commits

Folders and files

Repository files navigation

GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models

Paper Dataset Code

Overview

GACC-Bench evaluates goal-adaptive context compression: how well a language model preserves information needed for a high-level goal under a token budget, before the downstream questions are known.

  1. The compressor receives the full context, a goal, and a token budget.
  2. A fixed reader answers the goal's questions individually using the compressed context.
  3. A judge scores the answers against their references.

The benchmark contains 500 goals and 2,571 questions over 281 contexts, spanning seven document and agent-trace collections. Evaluation questions are withheld from the compressor.

News

  • [2026.10.17] The dataset is available on Hugging Face, with load_dataset() support, Parquet files, and original JSONL downloads.
  • [2026.10.15] Our paper is available on arXiv.

Main Results

Results on GACC-Bench-full (500 goals, 2,571 questions) from Table 2 of the paper. The models below serve as compressors; DeepSeek-V4-Pro is the fixed reader and judge. Budgets specify the percentage of source-context tokens retained.

All scores are accuracy percentages (higher is better). The seven source columns report Macro accuracy; the last two columns report overall Macro and Micro accuracy. Bold marks the best score in each column within a budget, including ties.

Scroll horizontally to view all columns.

5% retention budget

Compressor Dataset-level Macro Accuracy Overall Accuracy
LongBench-v2 HELMET L-Eval Loong MeetingBank LongCodeBench SWE-agent (SWE-Bench Verified) Macro Micro
GPT-6-Astra14.292.500.0011.760.006.4117.866.0042.43
GPT-6.1-Sol14.292.501.968.820.006.4110.715.4042.86
GPT-5.6-Sol6.122.501.961.960.001.280.001.8029.64
Claude-Opus-5.56.120.000.006.860.003.853.572.8028.94
Gemini-3.6-flash2.040.000.000.000.001.280.000.4020.30
Qwen3.8-Max6.122.501.960.980.006.4110.713.0030.84
GLM-5.314.290.001.967.840.005.1310.714.8031.19
GLM-5.3-flash8.160.000.984.900.001.287.142.6031.43
DeepSeek-V4-Pro8.162.500.000.000.005.133.572.0028.70
Kimi-K310.202.500.982.940.993.857.143.2028.67

10% retention budget

Compressor Dataset-level Macro Accuracy Overall Accuracy
LongBench-v2 HELMET L-Eval Loong MeetingBank LongCodeBench SWE-agent (SWE-Bench Verified) Macro Micro
GPT-6-Astra30.6112.5011.7621.570.9917.9517.8614.8057.41
GPT-6.1-Sol30.6112.509.8024.511.9814.1021.4314.8056.40
GPT-5.6-Sol14.290.005.885.881.985.133.575.2043.21
Claude-Opus-5.520.417.501.9614.711.9811.5417.869.2043.95
Gemini-3.6-flash18.372.500.001.960.005.137.143.6033.68
Qwen3.8-Max14.2915.002.944.900.996.4117.866.4044.96
GLM-5.320.417.507.8415.692.9712.8214.2910.8048.46
GLM-5.3-flash20.4115.003.9213.732.9710.2617.8610.0050.80
DeepSeek-V4-Pro10.205.001.964.900.0017.9510.716.2042.40
Kimi-K320.415.002.945.881.986.4117.866.6045.62

Metrics. Macro is the percentage of goals for which every associated question is answered correctly. Micro is the percentage of individual questions answered correctly. Overall Macro is computed over all 500 goals, rather than by averaging the seven source scores.

Dataset

Configuration Contexts Goals Questions
Full benchmark (default) 281 500 2,571
Small evaluation subset (testmini) 62 124 651

testmini contains two goals for each selected context and is a subset of the full benchmark. Both configurations use the test split.

Load from Hugging Face

pip install datasets
from datasets import load_dataset

dataset = load_dataset("Regimes/GACC-Bench", split="test")
mini = load_dataset("Regimes/GACC-Bench", name="testmini", split="test")

example = dataset[0]
print(example["goal"])
print(example["questions"][0]["question"])

Each row contains one goal, its full source context, and a list of questions with reference answers. The seven sources are HELMET, L-Eval, LongBench-v2, LongCodeBench, Loong, MeetingBank, and SWE-agent. Field descriptions and source-level statistics are in the dataset card.

Original JSONL files

The evaluator accepts the original files included in this repository:

The same files are hosted on Hugging Face under original/:

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="Regimes/GACC-Bench",
    repo_type="dataset",
    filename="original/GACC-Bench.jsonl",
    local_dir="hf_data",
)

Use the returned path as the evaluator's --input when downloading from Hugging Face.

Installation

git clone https://github.com/weiting24/GACC-Bench.git
cd GACC-Bench
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

The compressor, reader, and judge are configured independently using OpenAI-compatible APIs.

Evaluation

Replace the endpoint, key, and model placeholders with your own API settings:

bash run.sh \
  --input ./data/GACC-Bench.jsonl \
  --output ./runs/gacc_eval \
  --limit 0 \
  --budgets 5,10 \
  --include-full-context true \
  --compressor-url '<COMPRESSOR_URL>' \
  --compressor-key '<COMPRESSOR_KEY>' \
  --compressor-model '<COMPRESSOR_MODEL>' \
  --reader-url '<READER_URL>' \
  --reader-key '<READER_KEY>' \
  --reader-model '<READER_MODEL>' \
  --judge-url '<JUDGE_URL>' \
  --judge-key '<JUDGE_KEY>' \
  --judge-model '<JUDGE_MODEL>' \
  --workers 300 \
  --max-attempts 30 \
  --retry-sleep 30

Use --limit 10 and a separate output directory for a smaller run. Budgets are percentages of the original context length; 5,10 evaluates 5% and 10% conditions. The full-context option adds the uncompressed reference condition.

Metric Definition
Micro accuracy Fraction of questions answered correctly
Macro accuracy Fraction of goals for which every associated question is answered correctly

The evaluator writes reports/metrics.json, reports/results.jsonl, and reusable artifacts under memories/, answers/, and judgments/. Detailed output fields, retry behavior, and budget accounting are documented in the evaluation guide.

Dataset export

To reproduce the Hugging Face files from the checked-in JSONL:

pip install pyarrow
python scripts/export_huggingface.py --output ./release/huggingface

The exporter checks lossless Parquet round trips and subset membership, copies the original files byte-for-byte, and records source checksums in SOURCE.json. The dataset card template contains the split/configuration metadata.

Citation

If you use GACC-Bench, please cite:

@misc{liu2026gaccbench,
  title={GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models},
  author={Weiting Liu and Zhiyu Pan and Shuhong Wang and Zhenquan Yao and Jungang Li and Li Zhao and Jian Jiang and Jianfeng Feng and Ai Han and Wenlian Lu},
  year={2026},
  eprint={2610.21893},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2610.21893}
}

Star History

GACC-Bench Star History

About

No description, website, or topics provided.

Resources

Stars

78 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages