Skip to content

Commit 4f1dbc2

Browse files
authored
Merge pull request #1 from ymoslem/feat/telemath
TeleMath, cost-metric generalisation, and the Stage 2 cascade
2 parents 80a7b68 + 4a870db commit 4f1dbc2

23 files changed

Lines changed: 2228 additions & 70 deletions

.github/workflows/tests.yml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
name: tests
2+
3+
on: [push, pull_request]
4+
5+
jobs:
6+
test:
7+
runs-on: ubuntu-latest
8+
strategy:
9+
matrix:
10+
python-version: ["3.10", "3.12"]
11+
steps:
12+
- uses: actions/checkout@v4
13+
- uses: actions/setup-python@v5
14+
with:
15+
python-version: ${{ matrix.python-version }}
16+
cache: pip
17+
- run: pip install -e . pytest
18+
- run: pytest -q

REPRODUCE.md

Lines changed: 49 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,27 @@ The training-set cluster sizes used for the system-level accuracy and TPOT
4949
come from the paper's clustering (AIME train 194 / 405 / 322; TeleQnA train
5050
5,211 / 3,789).
5151

52+
### Stage 1+2 cascade latency (no GPU)
53+
54+
The combined Stage 1+2 system TPOT and E2EL are composed from the test-split
55+
per-cluster measurements plus the measured QE escalation counts, checked in
56+
under [`configs/`](configs):
57+
58+
```bash
59+
cre cascade --stats configs/aime_cascade_test.json
60+
cre cascade --stats configs/teleqna_cascade_test.json
61+
```
62+
63+
Each escalated query is charged both passes: for TPOT, per delivered token
64+
(`TPOT_strong + TPOT_eff * L_eff / L_strong`, following vLLM's per-request
65+
Mean TPOT convention); for E2EL, as the sum `E2EL_eff + E2EL_strong`, since
66+
Stage 2 inspects the complete efficient-model output before escalating. This
67+
gives 9.75 ms / 156,303 ms (AIME) and 23.65 ms / 1,127 ms (TeleQnA), matching
68+
the paper's Tables `aime_test` and `teleqna_test` Stage 1+2 latency (9.7 and
69+
23.8 ms) to within rounding. Expected values are pinned in
70+
[`tests/test_routing.py`](tests/test_routing.py). The escalated queries'
71+
accuracy recovery is measured separately (`cre qe-eval`, Appendix D).
72+
5273
### Reproducing the clustering
5374

5475
The first step in the paper's Stage 1 is to cluster the training queries. The released datasets already include the paper's clustering in the `cluster` column, so you can skip this step and use the released datasets directly. If you want to reproduce the clustering, you can run the following command:
@@ -114,6 +135,29 @@ cre qe-eval --classifier <checkpoint> --dataset ymoslem/AIME-clustered-output \
114135
(true / unnecessary / missed escalations) used in the QE appendices. For
115136
TeleQnA use `--max-length 512` and learning rate 2e-5.
116137

138+
### Building QE data for a new pool
139+
140+
For a pool other than the released ones, the QE data comes from the efficient
141+
model's own generations:
142+
143+
```bash
144+
# capture generations alongside the per-question outcomes
145+
cre evaluate ... --save-generations
146+
147+
# convert them to the schema cre qe-train reads
148+
python data/prep_qe.py --train <train_generations.jsonl> \
149+
--test <test_generations.jsonl> --out qe-data/<name>
150+
151+
# replay a trained classifier over the gated clusters
152+
cre qe-cascade --classifier <checkpoint> --generations <test_generations.jsonl> \
153+
--clusters 1,3 --strong-outcomes <strong_outcomes.jsonl> \
154+
--strong-model <name> --out configs/<pool>_cascade_test.json
155+
```
156+
157+
`--save-generations` adds the full outputs the classifier judges; per-question
158+
outcomes are always written. `cre qe-cascade` writes the per-cluster cascade
159+
accuracy and escalation counts into the cascade config that `cre cascade` reads.
160+
117161
## Serving the paper's pools
118162

119163
Two ready-made serving configs are provided:
@@ -142,15 +186,15 @@ Fetch any split as JSONL with `python data/download.py --dataset <id>`.
142186

143187
## Pinned environment
144188

145-
The exact environment used to produce the reported TPOT and accuracy numbers
146-
is pinned in [`requirements-paper.txt`](requirements-paper.txt) (vLLM 0.19.0,
147-
torch 2.10.0, Python 3.11, 2x A100 SXM 80 GB). This is a historical record, not
148-
a recommended version. TPOT is hardware- and version-specific and will shift on
189+
Package versions are pinned in
190+
[`requirements-paper.txt`](requirements-paper.txt). The reported numbers were
191+
measured on 2x A100 SXM 80 GB under Python 3.11 with 32 concurrent requests,
192+
averaged over 5 runs. TPOT is hardware- and version-specific and will shift on
149193
newer vLLM releases or different hardware (e.g. H100 with full W8A8 FP8
150194
support), which can also change the selected $\lambda^*$. Efficient ModernBERT
151195
training additionally used `flash-attn==2.8.3`.
152196

153-
Install order matters for the Gemma models: `pip install vllm==0.19.0` pulls
197+
Install order matters for the Gemma models: installing the pinned vLLM pulls
154198
transformers 4.57.6, which does **not** recognize the `gemma4` architecture.
155199
Upgrade with `pip install transformers==5.5.3` afterwards (it serves both the
156200
Qwen and Gemma pools; vLLM's `transformers<5` pin is conservative).

configs/aime_cascade_test.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"_comment": "AIME 2024 test set, Stage 1+2 cascade inputs. Per-cluster TPOT, E2EL and output length are measured on the test split (2xA100, vLLM, concurrency 32, 5-run mean). assignment is the lambda*=0.06 routing from `cre fit` on the training stats; escalations are the measured QE escalation counts (Table aime_qe_runs); cascade_accuracy is the measured per-cluster accuracy after the QE cascade on C1. `cre cascade` gives system accuracy 0.884, 9.75 ms TPOT and 156303 ms E2EL, matching the paper's Table aime_test (88.4%, 9.7 ms) to within rounding.",
3+
"cluster_sizes": {"0": 9, "1": 10, "2": 11},
4+
"assignment": {"0": "Qwen3-30B-A3B", "1": "VibeThinker-1.5B", "2": "Qwen3-30B-A3B"},
5+
"escalations": {"1": ["Qwen3-30B-A3B", 0.6]},
6+
"cascade_accuracy": {"1": 0.96},
7+
"models": {
8+
"VibeThinker-1.5B": {
9+
"errors": {"0": 0.311, "1": 0.100, "2": 0.291},
10+
"cluster_tpot_ms": {"0": 4.7212, "1": 4.7452, "2": 4.9764},
11+
"cluster_e2el_ms": {"0": 66051.5, "1": 67335.7, "2": 94304.2},
12+
"cluster_output_tokens": {"0": 13843.3, "1": 14168.8, "2": 18924.3}
13+
},
14+
"Qwen3-30B-A3B": {
15+
"errors": {"0": 0.133, "1": 0.020, "2": 0.171},
16+
"cluster_tpot_ms": {"0": 11.0254, "1": 11.7018, "2": 12.6006},
17+
"cluster_e2el_ms": {"0": 159115.8, "1": 150028.1, "2": 226696.6},
18+
"cluster_output_tokens": {"0": 14419.6, "1": 12809.3, "2": 17639.9}
19+
}
20+
}
21+
}

configs/teleqna_cascade_test.json

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"_comment": "TeleQnA test set, Stage 1+2 cascade inputs. Per-cluster TPOT, E2EL and output length are measured on the test split (2xA100, vLLM, concurrency 32, 5-run mean). assignment is the lambda*=0.07 routing from `cre fit` on the training stats; escalations are the measured QE escalation counts (Table teleqna_qe_runs); cascade_accuracy is the measured per-cluster accuracy after the QE cascade on C0. `cre cascade` gives system accuracy 0.743, 23.65 ms TPOT and 1127 ms E2EL, matching the paper's Table teleqna_test (74.3%, 23.8 ms) to within rounding.",
3+
"cluster_sizes": {"0": 590, "1": 410},
4+
"assignment": {"0": "Qwen3-4B", "1": "Gemma4-26B"},
5+
"escalations": {"0": ["Gemma4-26B", 202]},
6+
"cascade_accuracy": {"0": 0.740},
7+
"models": {
8+
"Qwen3-4B": {
9+
"errors": {"0": 0.311, "1": 0.360},
10+
"cluster_tpot_ms": {"0": 15.484, "1": 14.664},
11+
"cluster_e2el_ms": {"0": 663.93, "1": 696.12},
12+
"cluster_output_tokens": {"0": 39.5, "1": 43.8}
13+
},
14+
"Gemma4-26B": {
15+
"errors": {"0": 0.223, "1": 0.254},
16+
"cluster_tpot_ms": {"0": 24.565, "1": 24.412},
17+
"cluster_e2el_ms": {"0": 1206.13, "1": 1199.61},
18+
"cluster_output_tokens": {"0": 46.3, "1": 46.2}
19+
}
20+
}
21+
}

data/prep_gemma4_thinking.py

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env python
2+
"""Pre-render Gemma 4 chat-templated prompts with enable_thinking baked in.
3+
4+
Gemma 4's thinking switch is a chat-template kwarg (``enable_thinking``), not
5+
a text-level prefix like Qwen3's ``/no_think``. Its own chat_template.jinja
6+
(google/gemma-4-E2B-it) injects a ``<|think|>`` token at
7+
the top of the system turn only when ``enable_thinking`` is true; the model
8+
then opens its reply with ``<|channel>thought\\n...\\n<channel|>`` before the
9+
final answer. The model card confirms the same mechanism and notes that the
10+
E2B/E4B variants, unlike their larger siblings, emit no channel markers at
11+
all when thinking is disabled.
12+
13+
vLLM's own ``vllm bench serve`` applies the chat template itself before
14+
posting to ``/v1/completions`` (its ``CustomDataset.sample`` calls
15+
``tokenizer.apply_chat_template`` with a fixed set of keyword arguments), and
16+
that call never forwards a template kwarg such as ``enable_thinking``
17+
(verified by reading ``vllm/benchmarks/datasets.py``). So the switch has to
18+
be baked into the prompt text at prep time, here, with each row's fully
19+
rendered text stored as ``prompt``; the ``telemath_gemma4`` task entry in
20+
``evaluate.py`` sets ``pre_rendered=True`` so the benchmark passes
21+
``--skip-chat-template`` and serves the text verbatim.
22+
23+
Usage:
24+
25+
python data/prep_gemma4_thinking.py --in data/telemath_test.jsonl \\
26+
--out data/telemath_test_gemma --model google/gemma-4-E2B-it
27+
"""
28+
29+
from __future__ import annotations
30+
31+
import argparse
32+
import json
33+
from pathlib import Path
34+
35+
36+
def render_rows(rows: list[dict], tokenizer, enable_thinking: bool) -> list[dict]:
37+
"""Render each row's ``prompt`` through the model's own chat template.
38+
39+
The raw question is preserved under ``question`` before ``prompt`` is
40+
overwritten with the templated text, so the QE dataset built from these
41+
runs' generations carries the plain query, not chat-template markers.
42+
"""
43+
rendered = []
44+
for row in rows:
45+
text = tokenizer.apply_chat_template(
46+
[{"role": "user", "content": row["prompt"]}],
47+
add_generation_prompt=True,
48+
tokenize=False,
49+
enable_thinking=enable_thinking,
50+
)
51+
rendered.append({**row, "question": row.get("question", row["prompt"]), "prompt": text})
52+
return rendered
53+
54+
55+
def main() -> None:
56+
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
57+
parser.add_argument("--in", dest="in_path", required=True,
58+
help="an existing clustered prompts JSONL")
59+
parser.add_argument("--out", required=True,
60+
help="path prefix; _think.jsonl and _nothink.jsonl are appended")
61+
parser.add_argument("--model", default="google/gemma-4-E2B-it")
62+
parser.add_argument("--limit", type=int, default=0, help="0 means the whole file")
63+
args = parser.parse_args()
64+
65+
from transformers import AutoTokenizer
66+
67+
tokenizer = AutoTokenizer.from_pretrained(args.model)
68+
69+
rows = [
70+
json.loads(line)
71+
for line in Path(args.in_path).read_text().splitlines()
72+
if line.strip()
73+
]
74+
if args.limit:
75+
rows = rows[: args.limit]
76+
77+
out = Path(args.out)
78+
out.parent.mkdir(parents=True, exist_ok=True)
79+
for enable_thinking, suffix in ((True, "_think"), (False, "_nothink")):
80+
rendered = render_rows(rows, tokenizer, enable_thinking)
81+
path = out.with_name(f"{out.name}{suffix}.jsonl")
82+
with path.open("w", encoding="utf-8") as handle:
83+
for row in rendered:
84+
handle.write(json.dumps(row, ensure_ascii=False) + "\n")
85+
print(f"wrote {len(rendered)} rows to {path} (enable_thinking={enable_thinking})")
86+
87+
88+
if __name__ == "__main__":
89+
main()

data/prep_qe.py

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
"""Build a QE training dataset from ``*_generations.jsonl`` files.
2+
3+
Each generation row (written by ``cre evaluate --save-generations``) already
4+
carries everything the QE classifier needs -- ``prompt``, ``full_output``,
5+
``num_tokens`` and ``correct`` -- so this converter only relabels it into the
6+
schema ``cre qe-train`` expects: ``decision_label`` is 1 (accept) when the
7+
efficient model was correct, else 0 (route/escalate). It writes ``train.jsonl``
8+
and ``test.jsonl`` into an output directory that ``cre qe-train --dataset <dir>``
9+
loads directly, no Hugging Face Hub round-trip needed.
10+
11+
Usage:
12+
python data/prep_qe.py \
13+
--train tm_train_instruct_nothink_r5_..._generations.jsonl \
14+
--test tm_test_instruct_nothink_r5_..._generations.jsonl \
15+
--out data/telemath_router
16+
cre qe-train --dataset data/telemath_router --max-length 4096 --output-dir ./qe-telemath
17+
"""
18+
19+
from __future__ import annotations
20+
21+
import argparse
22+
import json
23+
from pathlib import Path
24+
25+
26+
def qe_row(gen: dict) -> dict:
27+
"""One generation row -> one QE example (columns match ymoslem/*-router)."""
28+
correct = bool(gen["correct"])
29+
full_output = gen["full_output"]
30+
return {
31+
"question": gen.get("question", gen.get("prompt", "")),
32+
"prompt": gen.get("prompt", ""),
33+
"ground_truth_answer": gen.get("ground_truth_answer"),
34+
"full_output": full_output,
35+
"answer": gen.get("answer"),
36+
"accuracy": float(correct),
37+
"num_words": len(full_output.split()),
38+
"num_tokens": gen["num_tokens"],
39+
"score": float(correct),
40+
"decision_label": 1 if correct else 0,
41+
"decision_str": "accept" if correct else "route",
42+
"cluster": gen.get("cluster"),
43+
"qid": gen.get("qid"),
44+
"run": gen.get("run"),
45+
}
46+
47+
48+
def to_qe_rows(generations: list[dict]) -> list[dict]:
49+
"""Convert generation rows to QE examples, pooling multiple files/models."""
50+
return [qe_row(g) for g in generations]
51+
52+
53+
def _read_jsonl(path: Path) -> list[dict]:
54+
with path.open() as f:
55+
return [json.loads(line) for line in f if line.strip()]
56+
57+
58+
def _write_jsonl(rows: list[dict], path: Path) -> None:
59+
path.parent.mkdir(parents=True, exist_ok=True)
60+
with path.open("w") as f:
61+
for r in rows:
62+
f.write(json.dumps(r, ensure_ascii=False) + "\n")
63+
64+
65+
def build(train_files: list[str], test_files: list[str], out_dir: str) -> dict[str, int]:
66+
"""Write ``{out_dir}/train.jsonl`` and ``test.jsonl``; return split sizes."""
67+
out = Path(out_dir)
68+
sizes = {}
69+
for split, files in (("train", train_files), ("test", test_files)):
70+
rows: list[dict] = []
71+
dropped = 0
72+
for f in files:
73+
gens = _read_jsonl(Path(f))
74+
# num_tokens feeds the QE input verbatim; a null (a generations file
75+
# written without output_lens) would render the string "None", so drop
76+
# those rows rather than poison the dataset.
77+
kept = [g for g in gens if g.get("num_tokens") is not None]
78+
dropped += len(gens) - len(kept)
79+
rows.extend(to_qe_rows(kept))
80+
if dropped:
81+
print(f"WARNING: dropped {dropped} {split} row(s) with null num_tokens")
82+
_write_jsonl(rows, out / f"{split}.jsonl")
83+
sizes[split] = len(rows)
84+
return sizes
85+
86+
87+
def main(argv: list[str] | None = None) -> None:
88+
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
89+
parser.add_argument("--train", nargs="+", required=True, help="generations JSONL file(s) for the train split")
90+
parser.add_argument("--test", nargs="+", required=True, help="generations JSONL file(s) for the test split")
91+
parser.add_argument("--out", required=True, help="output directory for train.jsonl / test.jsonl")
92+
parser.add_argument("--push-to-hub", default=None, help="also push the DatasetDict to this HF hub id")
93+
parser.add_argument("--hub-private", action="store_true")
94+
args = parser.parse_args(argv)
95+
96+
sizes = build(args.train, args.test, args.out)
97+
print(f"Wrote {args.out}/train.jsonl ({sizes['train']}) and test.jsonl ({sizes['test']})")
98+
label_pos = sum(
99+
1 for line in open(Path(args.out) / "train.jsonl") if json.loads(line)["decision_label"] == 1
100+
)
101+
print(f"Train accept/route balance: {label_pos} accept / {sizes['train'] - label_pos} route")
102+
103+
if args.push_to_hub:
104+
from datasets import load_dataset
105+
106+
ds = load_dataset("json", data_files={
107+
"train": str(Path(args.out) / "train.jsonl"),
108+
"test": str(Path(args.out) / "test.jsonl"),
109+
})
110+
ds.push_to_hub(args.push_to_hub, private=args.hub_private)
111+
print(f"Pushed to {args.push_to_hub}")
112+
113+
114+
if __name__ == "__main__":
115+
main()

0 commit comments

Comments
 (0)