[planetf1]

What changes for users

The container now starts with the 8b and 30b checkpoints given nothing but the checkpoint. Before, it stopped at startup: those checkpoints enable speech input, which makes granite-switch load a speech model while the engine starts, and the image runs with the Hub offline and did not contain that model. Decisions never use speech, so serve.sh now switches speech input off by default and no speech model is needed.

To keep speech input, set SYSTEMONE_SPEECH_INPUT=1 and make the speech model available. The README lists the ways: build with --build-arg WITH_ASR=1 (bakes it in), mount a Hub cache and set HF_HUB_CACHE, or set HF_HUB_OFFLINE=0. With the switch on and the model missing, serve.sh stops before vLLM with a message naming the model and these options.

The cost of the default is that /v1/chat/completions on the same server takes no audio input unless the switch is on.

docker build --build-arg WITH_ASR=1 also works again. It was failing at the step that downloads the speech model, because the model repository's main had moved off the commit the Dockerfile pins. The build now fetches the pinned commit, so it does not change when the model repository does.

Mechanism

• serve.sh passes --hf-overrides '{"asr_enabled": false}' unless SYSTEMONE_SPEECH_INPUT=1. Any value other than 0 or 1 is refused. The preflight check only runs when the switch is on.
• WITH_ASR stays 0 by default; the image holds no speech model unless built with WITH_ASR=1.
• The ASR step ran hf download of main and then required refs/main to equal ASR_REVISION. It now runs hf download --revision $ASR_REVISION. A download by commit writes no refs/main, and the offline lookup for revision main (which granite-switch uses) reads that file, so the step writes it and checks the pinned snapshot directory exists.
• README, docs/TESTING.md and docs/STATUS.md follow. tests/test_serve_sh.py runs serve.sh against a stub vllm.

Limitations

• Served under enroot, not docker run itself: its ENTRYPOINT, --gpus all handling and --ipc=host were not exercised.
• Speech input on was tested on the 8b with the model from a mounted cache. The WITH_ASR=1 image (model baked in) was served on the 8b earlier, before the switch existed, and the 30b was not run with speech input on.
• The cause is upstream: granite-switch loads the speech model even when no audio request can arrive. This works around it.
• No release is planned for this change.

Testing

• The default and WITH_ASR=1 images build (linux/amd64).
• Default image, through serve.sh, only the checkpoint mounted, Hub offline: the 8b on one H100, and the 30b on two H100s with --tensor-parallel-size 2. On each, e2e_check 11/11, probe_check 6/6 and accuracy_check pass. The server log shows the override applied and never names the speech model.
• SYSTEMONE_SPEECH_INPUT=1 with the speech model from a mounted cache: the 8b starts and passes the same checks.
• Negative control, SYSTEMONE_SPEECH_INPUT=1 with no speech model anywhere: serve.sh exits 1 with the preflight message and vLLM never starts.
• tests/test_serve_sh.py and tests/test_preflight.py pass; ruff and the pre-commit hooks pass.