Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pr_quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ jobs:
- name: Launcher contracts
run: |
pytest -q \
tests/test_launch_aurora.py \
tests/test_launch_aurora_web_vla.py \
tests/platform/test_perlmutter_launch_contract.py \
tests/platform/test_hpc_bridge_contract.py \
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ will make up v0.1.0.
per-modality support-status table
- `tools/ci/check_doc_links.py`, a link checker for relative Markdown links, run
as a required `doc_links` CI job
- `docs/tutorials/`, an index of end-to-end tutorials grouped by application
area. The materials tutorial moved from `docs/materials_prism_tutorial.md` to
`docs/tutorials/material_science/materials_prism_tutorial.md`; notebooks stay
next to their scripts in `applications/<area>/`

### Changed

Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,7 @@ descriptions and return types where they are not obvious from the signature.
| `modalities/` | Encoders, projectors, output decoders |
| `models/` | Backbone integrations |
| `evaluation/` | Universal evaluator, vLLM inference |
| `tutorials/` | End-to-end walkthroughs, one folder per application area; notebooks stay in the top-level `applications/<area>/`, not `docs/applications/` |
| `applications/` | Scientific applications (BioReason, KEGG, materials) |
| `api/` | API reference for the public `src.*` surface |
| `development/` | CI and contributor tooling |
Expand Down
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ multimodal: ## Run the contract suite as CI's `multimodal_contract` job does
# none of them -- this target is the only make path that does.
launcher: ## Run the launcher contracts as CI's `launcher_contract` job does
$(PYTHON) -m pytest -q \
tests/test_launch_aurora.py \
tests/test_launch_aurora_web_vla.py \
tests/platform/test_perlmutter_launch_contract.py \
tests/platform/test_hpc_bridge_contract.py \
Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,7 @@ Per-platform detail, including lockfile regeneration:
| **Text** | [OLMo-1B / OLMo-3 7B](https://huggingface.co/allenai/OLMo-7B-0724-hf) | 2048 / 4096 |
| **Image** | [SigLIP2](https://huggingface.co/google/siglip2-base-patch16-224) | 768 |
| **Time series** | [Moirai 2.0](https://huggingface.co/Salesforce/moirai-2.0-R-small) / [Intern-S2 preview](https://huggingface.co/internlm/Intern-S2-Preview-397B) / [TimeOmni](https://arxiv.org/abs/2510.03255) | Variable by backend |
| **Spectrum** | [AION-1 Base](https://huggingface.co/polymathic-ai/aion-base) | 768 |
| **DNA** | [Nucleotide Transformer v2 500M](https://huggingface.co/InstaDeepAI/nucleotide-transformer-v2-500m-multi-species) | 1024 |
| **Geometry** | [Walrus](https://huggingface.co/polymathic-ai/walrus) | 512 |
| **Graph** | GraphMAE2 (PyG) | 768 |
Expand All @@ -193,10 +194,11 @@ actually exercised by shipped configs and CI, not what the architecture permits.

| Modality | Status | Shipped model configs | Notes |
|---|---|---|---|
| **Text** | Supported | 26 of 26 | Every config enables it. With an HF backbone, the backbone's own embeddings are used and `TextEncoder` is bypassed. |
| **Image** | Supported | 9 of 26 | Best-validated multimodal path; the basis of the scaling results below. |
| **Time series** | Supported | 14 of 26 | Five encoder backends (`linear`, `moirai`, `intern_s2`, `intern_s2_397b`, `timeomni`). One of two modalities with a vLLM inference processor (the other is image). |
| **DNA** | Supported | 2 of 26 | Nucleotide Transformer encoder (`src/encoders/dna.py`), used by the BioReason/KEGG path — see [docs/applications/bioreason_grpo.md](docs/applications/bioreason_grpo.md). Input-only. |
| **Text** | Supported | 29 of 29 | Every config enables it. With an HF backbone, the backbone's own embeddings are used and `TextEncoder` is bypassed. |
| **Image** | Supported | 11 of 29 | Best-validated multimodal path; the basis of the scaling results below. |
| **Time series** | Supported | 14 of 29 | Five encoder backends (`linear`, `moirai`, `intern_s2`, `intern_s2_397b`, `timeomni`). One of two modalities with a vLLM inference processor (the other is image). |
| **DNA** | Supported | 2 of 29 | Nucleotide Transformer encoder (`src/encoders/dna.py`), used by the BioReason/KEGG path — see [docs/applications/bioreason_grpo.md](docs/applications/bioreason_grpo.md). Input-only. |
| **Spectrum** | Supported | 2 of 29 | AION-1 Base codec + shared encoder (`src/encoders/aion_spectrum.py`), used by the AstroPRISM path — see [docs/tutorials/astroprism/astroprism_tutorial.md](docs/tutorials/astroprism/astroprism_tutorial.md). Input-only. |
| **Graph** | Experimental | 0 | Requires `torch_geometric`. Runs in the per-modality sweep (172.7 samp/s) but no shipped model config enables it, and the handler is labelled untested in its own smoke config. |
| **Table** | Experimental | 0 | TAPAS-based; runs in the sweep (65.5 samp/s). Input-only — no output decoder. No shipped model config enables it. |
| **Geometry** | Experimental | 0 | Requires `walrus` and `the_well`. Runs in the sweep (69.4 samp/s) **on synthetic data** — no real PDEBench corpus is staged, and `GeometryEncoder` hardcodes `input_dim=6`. A `WALRUS_FALLBACK=1` stub exists for smoke runs and produces meaningless features. |
Expand Down Expand Up @@ -375,6 +377,7 @@ PRISM/
[docs/index.md](docs/index.md) indexes every page. Most-used entry points:

- [docs/getting-started.md](docs/getting-started.md) — clone → environment → first run
- [docs/tutorials/README.md](docs/tutorials/README.md) — end-to-end tutorials by application area (BioReason, materials, …)
- [docs/platforms/site_paths.md](docs/platforms/site_paths.md) — configuring the `PRISM_*` roots
- [docs/platforms/aurora_operations.md](docs/platforms/aurora_operations.md) — Aurora ops, env, CCL tuning, troubleshooting
- [docs/training/training.md](docs/training/training.md) — training stages
Expand Down
27 changes: 27 additions & 0 deletions docs/applications/astroprism.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# AstroPrism: the 646-candidate pilot

Start with the **646 positional matches between released paired AION galaxies and AION-Search image captions**. Recover and validate these objects on Aurora, train image-to-language caption alignment, prepare grounded spectral captions in parallel, then extend to image + spectrum + language, instruction SFT, and a later verifier-based RL experiment.

The earlier goal of **10,000 training galaxies plus separate validation/test galaxies** remains a later expansion after this small pipeline works.

## Contents

| File | Purpose |
|---|---|
| [astroprism_training_plan.md](astroprism_training_plan.md) | AION-1/Qwen architecture, phases, splits, training settings and implementation work |
| [astroprism_synthetic_data_prompts.md](astroprism_synthetic_data_prompts.md) | Image verification, spectral facts/captions, joint captions, instruction conversations and validation prompts |
| [astroprism_benchmark_landscape.md](astroprism_benchmark_landscape.md) | AstroSpecQA, image benchmarks, paired/interleaved evaluation and RL evaluation boundaries |
| [astroprism_data.md](astroprism_data.md) | Dataset pins, roster checksum, field definitions and remote recovery procedure |
| [astroprism_data/overlap_candidates_646.csv.gz](astroprism_data/overlap_candidates_646.csv.gz) | Exact 646-row candidate roster with IDs, coordinates, separations and source shard/row locators |

This page and the four it links to are the whole package. It is a **plan**: no AION inference, no astronomy-specific PRISM integration, no caption synthesis, no SFT and no RL is implemented here. The analysis scripts, audit tables and diagnostic arrays behind the roster are not part of it.

## What is ready

The positional-match roster is committed here. Its 646 paired catalog IDs all map uniquely into the official tutorial raw-file index, with matching DESI IDs for these candidates. Physical association, raw observation quality and compatibility of each caption with AION's exact image crop still need checking. Caption text has not yet been extracted for this roster.

The proposal uses AION-1 Base's modality-specific codecs and shared frozen encoder, separate image/spectrum projectors, and Qwen3-4B. Nothing here establishes model readiness; the numbers below are proposals, not measurements.

For a 646-object accepted pool, the proposed split is approximately **516 train / 65 validation / 65 test**, grouped by physical source before training or instruction generation. Rejections or grouping can change those counts. The 646 rows are the complete candidate pool, not 646 training objects plus additional holdouts.

Remote data recovery, model loading, job submission and training are all subsequent work. [astroprism_data.md](astroprism_data.md) documents how to recover the captions and raw observations from the pinned public sources.
Loading
Loading