Skip to content
UniPat-AIPublic

About

Science Prediction Universe (SPU): a benchmark of hidden wet-lab outcomes. Community task submissions and review.

Resources

Contributing

Stars

13 stars

Watchers

0 watching

Forks

Repository files navigation

Science Prediction Universe #Benchmark, UniPat AI

Science Prediction Universe (SPU)

Paper · Homepage · Dataset · Blog

An SPU task: the experimental definition goes to a wet lab and to an LLM agent; the agent's predicted ranking is scored against the hidden measured ranking by Spearman correlation.

Science Prediction Universe (SPU) is a benchmark of hidden wet-lab outcomes. Each task fixes the candidates, materials, conditions and measurement protocol of a real experiment, and an agent ranks the candidates by the measured outcome. Rankings are scored by Spearman correlation against the measurements.

SPU has 100 public tasks, built from 85 papers in 11 Nature Portfolio journals across chemistry, materials science, biology and medicine, and 16 private tasks from unpublished wet-lab experiments. The private tasks are not released.

Coverage of SPU: the 116 tasks by discipline and subfield, their source journals and the number of experiments per task.

Results

Thirteen agent configurations, each run once on every task with an eight-hour budget, rank the measured outcomes far from perfectly: the best mean is below 0.5 on a scale where a perfect ranking scores 1 and random rankings average 0.

Thirteen agent configurations ranked by mean Spearman score on all scored tasks, with 95% bootstrap intervals.

Means over the public and private tasks on which each configuration has a scored run, with 95% bootstrap intervals over tasks. Claude Opus 5.5 and Claude Fable 5.1 are scored on fewer tasks because most of their runs ended in safety refusals. Hatched bars are means on the 28 tasks that all thirteen configurations completed, and the right-hand column ranks the configurations on those tasks. See the paper for task-adjusted means, paired tests and cost.

Run the benchmark

The 100 public tasks are released as complete Harbor tasks on Hugging Face, with measurements, graders and oracle solutions:

hf download UnipatAI/SPU --repo-type dataset --local-dir SPU && cd SPU
harbor run -p tasks/fluorene-dehydration-concentrations -a oracle -e docker \
  --override-gpus 0 --override-cpus 1 --override-memory-mb 1024 -o ../jobs

The dataset card describes the run budget used in the paper, scoring and how to report results.

What this repository is for

This repository is where the community contributes new tasks and reports problems:

  • New task: open a pull request with a detailed task plan, a reference ranking workbook and supporting data under submissions/<task-id>/. See CONTRIBUTING.md and the templates.
  • Correction to a task: open a task issue, then a fix pull request.
  • Unpublished or restricted experiments: do not open a PR. Use the private workspace on the SPU project site.

Every task PR gets two automatic checks: spu/pr-format (title and description, see docs/PR-FORMAT.md) and spu/clarity-audit (an AI completeness review of the plan and data, see docs/REVIEW.md). These check completeness, not scientific validity; maintainers review every task before acceptance.

The project site and its services are developed separately.

Authors

Project Team: Zengji Tu1*, Xiang Liu1*, Jiarui Sun1, Ruoyu Wu1, Zhihui Qi1, Chengda Wu1, Yutao Zhou1, Bo Yang1, Ziyan Zhang1, Jialong Wu1, Liang Chen1†‡, Kuan Li1†

Advisory Committee: Steven Dillmann2, Yuanqi Du3, Soojung Yang4, Shizhe Diao5, Jianjun Wu6, Junlang Liu7, Xuming Hu8, Lin Chang6,9†, Baobao Chang10, Xiaowen Chu8, Philip S. Yu11, Tong-Yi Zhang8,12

Wet-Lab Collaborators: Hao Cheng13, Qin Han13, Yanye Feng14, Chenru Duan15, Chuncheng Liu15, Yan Liu15, Yuchen Gao15, Zhao Li15, Ronghua Zhang15, Tory Xu15, Renjian Song16, Yaokai Fu16

*Core contributors. †Corresponding authors: Liang Chen (liangchen@unipat.ai), Kuan Li (kuanli@unipat.ai), Lin Chang. ‡Project leader: Liang Chen.

Affiliations
  1. UniPat AI
  2. Stanford University
  3. Cornell University
  4. FutureHouse
  5. The Hong Kong University of Science and Technology
  6. State Key Laboratory of Photonics and Communications, School of Electronics, Peking University
  7. Department of Chemistry and Chemical Biology, Harvard University
  8. The Hong Kong University of Science and Technology (Guangzhou)
  9. Frontiers Science Center for Nano-optoelectronics, Peking University
  10. Peking University
  11. University of Illinois Chicago
  12. Shanghai University
  13. Yurogen Biosystems LLC
  14. ABclonal Biotechnology Co., Ltd.
  15. Deep Principle
  16. Bota Biosciences

Citation

@misc{spu2026,
  title         = {Science Prediction Universe},
  author        = {Tu, Zengji and Liu, Xiang and Sun, Jiarui and Wu, Ruoyu and Qi, Zhihui and Wu, Chengda and Zhou, Yutao and Yang, Bo and Zhang, Ziyan and Wu, Jialong and Chen, Liang and Li, Kuan and Dillmann, Steven and Du, Yuanqi and Yang, Soojung and Diao, Shizhe and Wu, Jianjun and Liu, Junlang and Hu, Xuming and Chang, Lin and Chang, Baobao and Chu, Xiaowen and Yu, Philip S. and Zhang, Tong-Yi and Cheng, Hao and Han, Qin and Feng, Yanye and Duan, Chenru and Liu, Chuncheng and Liu, Yan and Gao, Yuchen and Li, Zhao and Zhang, Ronghua and Xu, Tory and Song, Renjian and Fu, Yaokai},
  year          = {2026},
  eprint        = {2611.15049},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2611.15049},
  note          = {Project page: https://spu.unipat.ai}
}

License

Code in this repository is licensed under the Apache License 2.0. The SPU dataset on Hugging Face is released under CC BY 4.0. UniPat brand assets are not covered by either license.

Contact: Liang Chen (liangchen@unipat.ai), Kuan Li (kuanli@unipat.ai).

About

Science Prediction Universe (SPU): a benchmark of hidden wet-lab outcomes. Community task submissions and review.

Resources

Contributing

Stars

13 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages