This repository is the official implementations of "Off-Policy Evaluation of Large Language Models via Learned Semantic Bottleneck Embeddings"(NeurIPS 2026). The following workflow runs an experiment from response generation to the final policy-value estimates.
The example uses 100 HelpSteer2 prompts, Llama-3.2-1B-Instruct as the behavior policy, Qwen2.5-7B-Instruct as the target policy, and Skywork-Reward-V2-Qwen3-8B as the reward model. Run all commands from the directory containing this README.
Use a fresh Python 3.10 environment on a Linux CUDA host for generation and reward scoring:
conda create -n sbe-mips python=3.10 pip -y
conda activate sbe-mips
python -m pip install --upgrade pip
python -m pip install -r requirements-generation.txt
python -m pip install --no-cache-dir --no-build-isolation --no-deps flash-attn==2.8.3
python -m pip check
python -c "import torch, flash_attn; print(torch.__version__, flash_attn.__version__)"requirements-generation.txt includes vLLM and all training dependencies. It pins vLLM 0.11.0 and PyTorch 2.8.0 so pip resolves them together. Install this file directly; do not first install the training requirements or separately upgrade vLLM/PyTorch. The prebuilt vLLM wheel requires a compatible NVIDIA driver; see the vLLM installation documentation.
Skywork scoring uses FlashAttention 2. Install flash-attn==2.8.3 after PyTorch as shown above; --no-build-isolation builds against that installation, and --no-deps preserves the resolved GPU stack. --no-cache-dir avoids cross-device wheel moves when the pip cache is on a different filesystem. Its build and runtime dependencies are included in requirements-generation.txt. Source builds require a matching CUDA toolkit with nvcc; FlashAttention 2 supports Ampere, Ada, and Hopper GPUs with FP16/BF16. See the FlashAttention installation instructions.
Model downloads require internet access; gated models require Hugging Face authentication and accepted licenses. For training and evaluation only, install requirements.txt in a separate environment; vLLM is not required.
| Model | License |
|---|---|
| meta-llama/Llama-3.2-1B-Instruct | Llama 3.2 Community License (llama3.2) |
| Qwen/Qwen2.5-7B-Instruct | Apache License 2.0 (apache-2.0) |
| google/gemma-3-1b-it | Gemma Terms of Use (gemma) |
| google/gemma-3-12b-it | Gemma Terms of Use (gemma) |
| Skywork/Skywork-Reward-V2-Qwen3-8B | Apache License 2.0 (apache-2.0) |
| OpenAssistant/reward-model-deberta-v3-large-v2 | MIT License (mit) |
| RLHFlow/ArmoRM-Llama3-8B-v0.1 | Meta Llama 3 Community License (llama3) |
| Alibaba-NLP/gte-modernbert-base | Apache License 2.0 (apache-2.0) |
Generate responses from both policies on the same 100 prompts:
python generate_data.py \
--config configs/data_generation/helpsteer_Llama-3.2-1B-Instruct.yaml \
--run-id hsteer100_llama \
--set dataset.n=100
python generate_data.py \
--config configs/data_generation/helpsteer_Qwen2.5-7B-Instruct.yaml \
--run-id hsteer100_qwen \
--set dataset.n=100Each policy generates ten responses per prompt, giving 1,000 responses per policy. Sampling uses temperature 0.7, top-p 0.85, and a maximum of 256 new tokens. Prompts are deduplicated and filtered to 250 tokens under the DeBERTa tokenizer before the first 100 are selected.
The generated datasets are saved to:
data/hsteer100_llama/logged_dataset.parquet
data/hsteer100_qwen/logged_dataset.parquet
Apply Skywork to both generated datasets, using the same configurations and run IDs:
python score_rewards.py \
--config configs/data_generation/helpsteer_Llama-3.2-1B-Instruct.yaml \
--run-id hsteer100_llama
python score_rewards.py \
--config configs/data_generation/helpsteer_Qwen2.5-7B-Instruct.yaml \
--run-id hsteer100_qwenSkywork is the only enabled reward model. RM-DeBERTa and ArmoRM remain commented out in the configurations.
The scored files used in the next step are:
results/hsteer100_llama/hsteer_Llama-3.2-1B-Instruct.parquet
results/hsteer100_qwen/hsteer_Qwen2.5-7B-Instruct.parquet
Each row contains a prompt, a generated response, its generation index, and the reward/skywork score. Existing generated responses can be reused: rerun this scoring stage with the same run IDs, then convert the Skywork rewards into a new prepared-data directory and train new checkpoints.
First calculate the frozen GTE embeddings for the prompts and responses from both policies:
python calculate_embeddings.py \
--input results/hsteer100_llama/hsteer_Llama-3.2-1B-Instruct.parquet \
results/hsteer100_qwen/hsteer_Qwen2.5-7B-Instruct.parquet \
--model Alibaba-NLP/gte-modernbert-base \
--dataset hsteer --device cuda --output data/hsteer100This writes a shared embedding bank and its text index under data/hsteer100/embeddings/. Prompt and response texts are embedded separately. Change --model to another compatible Hugging Face model ID to use a different embedding encoder.
Then match the behavior and target datasets by prompt and convert them into training pairs:
python convert_to_npz.py \
--behavior results/hsteer100_llama/hsteer_Llama-3.2-1B-Instruct.parquet \
--target results/hsteer100_qwen/hsteer_Qwen2.5-7B-Instruct.parquet \
--dataset hsteer --reward skywork --seed 0 --output data/hsteer100For each prompt, conversion independently selects one behavior response and one target response. The resulting pair contains their embedding indices and the observed behavior reward. All responses belonging to a prompt remain in the same partition.
Conversion uses a 70/15/15 prompt split, with 10% of the training portion reserved for monitoring. The current fixed-epoch trainer does not use monitoring rewards to select epochs. Target rewards are excluded from fitting and monitoring; the evaluation reference averages all ten target rewards for each held-out prompt.
The prepared directory contains:
data/hsteer100/
embeddings/ # Shared .npy bank and text index
tasks/ # NPZ training pairs and split metadata
tasks.csv # Task names and partition sizes
task_configs.json # Training settings for each task
Keep these files together. The NPZ arrays reference rows of the shared embedding bank. Training settings are copied from configs/training.json; edit that file before conversion, or pass --template PATH.
python train.py \
--data data/hsteer100 \
--device cuda \
--output trained_checkpoints/hsteer100This trains DM, MIPS, reward-supervised MIPS, and SBE-MIPS using the prepared pairs. Checkpoints are written to trained_checkpoints/hsteer100/. Use --device cpu for CPU training.
python evaluate.py \
--data data/hsteer100 \
--checkpoints trained_checkpoints/hsteer100 \
--output results/hsteer100/predictions.parquetEvaluation runs on the validation and test partitions and produces two files:
| File | Contents |
|---|---|
results/hsteer100/predictions.parquet |
Per-prompt estimator contributions and rewards |
results/hsteer100/predictions.summary.csv |
Policy-value estimates and squared errors for each method and partition |
Display the final test results in Python:
import pandas as pd
results = pd.read_csv('results/hsteer100/predictions.summary.csv')
print(results.loc[results['source'] == 'test',
['method', 'n_prompts', 'estimate', 'target_value', 'squared_error']]
.to_string(index=False))estimate is the estimated target-policy value. target_value is the mean reward of the cached target responses on the same held-out prompts. squared_error is their squared difference. Each run reports one squared error per estimator; MSE requires averaging these errors across repeated runs. The 100-prompt example demonstrates the complete workflow on a small dataset.
For all available prompts, omit --set dataset.n=100 in both generation commands. Use new run IDs and prepared-data/checkpoint directories when changing the experiment. Unchanged generation and reward-scoring jobs can resume; an existing embedding bank is reused by proceeding directly to conversion rather than recalculating it.
The configuration configs/data_generation/helpsteer_Gemma3-1b-it.yaml, provides Gemma-3-1B-it. To use it as either policy, generate and score its responses, include its scored file in embedding calculation, and pass it as --behavior or --target during conversion.
New embeddings are calculated with mean pooling, L2 normalization, and maximum length 512, then saved directly to a .npy bank with a JSON text index. No database is required.
Generate and score 100 UltraFeedback prompts with Llama-3.2-1B-Instruct and Skywork:
python generate_data.py \
--config configs/data_generation/ultrafeedback_Llama-3.2-1B-Instruct.yaml \
--run-id ufb100_llama --set dataset.n=100
python score_rewards.py \
--config configs/data_generation/ultrafeedback_Llama-3.2-1B-Instruct.yaml \
--run-id ufb100_llamaThe scored output is results/ufb100_llama/ufb_Llama-3.2-1B-Instruct.parquet. Use --dataset ufb for embedding calculation and NPZ conversion. Policy-transfer evaluation also requires responses from a target policy on the same UltraFeedback prompts.
sample_data/ contains one small HelpSteer2 task for Gemma-3-1B → Gemma-3-12B with Skywork rewards and real cached GTE embeddings. To start directly at training and evaluation:
python train.py --data sample_data --device cpu --output trained_checkpoints/samples
python evaluate.py --data sample_data --checkpoints trained_checkpoints/samples \
--output results/samples/predictions.parquetTraining hyperparameters are defined in configs/training.json and recorded for each prepared task in task_configs.json. Split IDs and sampling seeds are saved alongside the task archives. The implementation uses product/difference pair features, an additive context baseline, and a representation stage followed by classifier-odds fitting. Target actions are sampled during conversion and remain fixed during training.