A toolkit for evaluating Jev-like decision models.
English · 简体中文
Documentation · Leaderboard · Datasets
Introduction · News · Quick start · Models · Benchmarks · Results · Local guides
JevalKit is an open-source evaluation toolkit for Jev-like decision models, providing unified model interfaces and prepared evaluation datasets. Users can evaluate models across benchmarks with a single command, without preparing data in separate repositories.
Integration requests: Simply open an issue with the model or benchmark ID, and we can help add support for it.
- 2026-10-19: Released the 🤗 JevalKit Leaderboard with benchmark and model-size filters, model comparison, probability metrics and Markdown, TSV and JSON result exports. Complete evaluations, smoke tests and imported predictions are distinguished.
- 2026-10-18: Integrated eight benchmarks: VisualDecisionBench, JevBench, Decision Index (public components), Typed-Decisions, DecisionBench, Procedural Typed Decisions, zh-decision-bench and JevAlt-Bench. Prepared evaluation datasets are available in 🤗 JevalKit Datasets.
- 2026-10-18: Added support for 20 Jev-like model families, with real inference verified for 18. Hosted API adapters for Jev and OpenAI Decisions are implemented; live evaluations remain pending.
git clone https://github.com/Valen-Team/JevalKit.git
cd JevalKit
python -m pip install -e '.[hf,data]'The CLI and scorers require Python 3.10+. Model inference uses the dependencies specified by each author; some models require Python 3.12+ or a separate serving environment. See the model integration guide.
export JEVALKIT_DATA_ROOT="$PWD/data"
hf download Valen-Team/Valen-0.8B \
--local-dir "$JEVALKIT_DATA_ROOT/models/Valen-0.8B"
hf download Valen-Team/JevalKit_Datasets --repo-type dataset \
--include "datasets/JevBench.tsv" --local-dir "$JEVALKIT_DATA_ROOT"The published TSVs are ready to evaluate. To download all published views, replace
datasets/JevBench.tsv with datasets/*.tsv. Visual media and Decision Index
scoring inputs have separate downloads in the usage guide.
python -m jevalkit run --model Valen-0.8B --data JevBench \
--option model="$JEVALKIT_DATA_ROOT/models/Valen-0.8B" \
--output "$JEVALKIT_DATA_ROOT/results"
python -m jevalkit leaderboard "$JEVALKIT_DATA_ROOT/results" \
--output "$JEVALKIT_DATA_ROOT/leaderboard"Pass --option model=/path/to/checkpoint to reuse existing weights. Use
--limit 10 for a smoke test, or --resume to continue an interrupted run with
the same settings. Full usage guide →
Size variants and Clef/Clef-Flash count within their respective families. Measured means real inference has been verified for at least one registered variant; per-benchmark coverage is shown in the leaderboard.
View supported models (20 families)
| Model family | Registered model IDs | Interface | Evaluation |
|---|---|---|---|
| Valen | Valen-0.8B, Valen-2B, Valen-4B |
Native | Measured |
| Valen API | Valen-API-0.8B, Valen-API-2B, Valen-API-4B |
Hosted API | Measured on delivery_support |
| Intern-Decision | Intern-Decision-0.8B, Intern-Decision-2B, Intern-Decision-4B |
Native | Measured |
| Laya | Laya |
Native | Measured |
| Liquid d1 | Liquid-d1-3B |
Native | Measured |
| vLLM Decision1 | Decision1-Kai |
Native | Measured |
| Clef | Clef, Clef-Flash |
Native | Measured |
| Pplx-Decider | Pplx-Decider |
Native | Measured |
| JevK5 | JevK5 |
Native | Measured |
| StartLux | StartLux-0.8B, StartLux-v1.1-{0.8B,2B,4B,9B,27B}-Instant |
Native / Author server | v1 measured; v1.1 pending |
| Crivo | Crivo-150M |
Native | Measured |
| RSI-Jev | RSI-Jev-v4 |
Native | Measured |
| Vjev | Vjev-4B |
Native | Measured |
| QEV | QEV |
Native | Measured |
| OmniJev | OmniJev |
Native | Measured |
| Visual-Jev | Visual-Jev |
Native | Measured |
| Kev | Kev |
Author server | Measured |
| H2O-Lightning | H2O-Lightning |
Author server | Measured |
| OpenJev | OpenJev |
Author server | Measured |
| Jev | Jev |
Hosted API | Full evaluation available |
| OpenAI Decisions | OpenAI-Decisions |
Hosted API | Live evaluation pending |
The catalog also includes a Uniform diagnostic baseline and general LLM baselines (LLM-OpenAI, LLM-Anthropic, LLM-Gemini) for comparing decision models with LLMs on agreement, latency, tokens and estimated cost; the same comparison runs inside applications through the Python SDK's Valen.llm. Author dependencies,
context limits, supported question types and media contracts are documented in
the integration guide.
View supported datasets (8 benchmark suites)
| Benchmark | Input | Evaluation data | Main measurements |
|---|---|---|---|
| VisualDecisionBench | 🖼️ Image · 🎬 Video | 1,696 states; 9,143 questions | Accuracy, soft-target Brier/NLL, ordinal metrics |
| JevBench | 📝 Text | 231 public tasks across easy/original/hard tiers | Official task scores and calibration |
| Decision Index | 📝 Text · 🖼️ Image | 0.4.1 public tracks: 140,178 text requests; 12,210 vision questions | Official weighted public text and vision indices |
| Typed-Decisions | 📝 Text | 400 states; 2,000 decisions | Modal accuracy, KL and soft-target Brier |
| DecisionBench | 📝 Text | 23,900 evaluation rows; 43 tasks | Accuracy, reference NLL and calibration |
| Procedural Typed Decisions | 📝 Text | 18,000 test states; 57,016 questions | Rule execution and exact probability judgments |
| zh-decision-bench | 📝 Chinese | v0.2: 378 items; 467 questions | Routing, moderation, ordinal ratings and calibration |
| JevAlt-Bench | 📝 EN · TR · DE | 3,116 test states; 8,434 questions | Labeled and soft-probability decisions; language slices |
VisualDecisionBench's image and video views belong to the same benchmark. Decision Index public scores and its private leaderboard components are reported separately. See the dataset packages and scoring protocols.
Download the prepared TSVs from 🤗 JevalKit Datasets. Use the matrix example to evaluate several models and benchmarks.
The results summary recorded on 2026-10-27 contains 214 eligible full scores and 300 retained run artifacts, covering 19 measured model families; OpenAI-Decisions remains pending. The leaderboard is a separately published snapshot and may differ. 🤗 Open the leaderboard.
Read the JevalKit documentation on the website, or use the repository guides below.
| Guide | Contents |
|---|---|
| Running evaluations | Downloads, evaluation commands, resume, APIs and parallel runs |
| Model integrations | Author SDKs, runtime dependencies and input limits |
| Scoring protocols | Dataset definitions, probability metrics and official scorers |
| Workflows, cost and weak slices | End-to-end workflows with decisions in code, LLM cost, slice analysis |
Each model implementation has its own file in jevalkit/models/, registered in
jevalkit/config.py. Each benchmark has its own directory
in jevalkit/datasets/, containing preparation, scoring and frozen TSV metadata.
Pull requests are welcome from anyone interested in contributing to the project.
Bug reports and contributions are welcome through Issues and Pull Requests.
 Liuziyu77 |
 yph22 |
 SYuan03 |
 yuhangzang |
JevalKit is released under Apache-2.0. We thank VLMEvalKit for its evaluation toolkit design, and the authors of the integrated decision models and benchmarks.
Reuse per-question predictions offline, fit and save temperatures, freeze automation thresholds on separate validation data, then measure held-out coverage and accepted error. Reports include before/after NLL, Brier, ECE, reliability curves and failure counts, without changing official scoring. See the workflow and cache protocol. Fitting and replay need no model downloads.