feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support, and language_options for Whisper STT by A-K-Erol · Pull Request #6700 · livekit/agents

A-K-Erol · 2026-08-04T22:25:14Z

Summary

Baseten hosts Qwen3-ASR Streaming and Qwen3-TTS alongside the Whisper and Orpheus models this plugin already supports. They speak different wire protocols, so the existing STT/TTS classes can't reach them — pointing either at a Qwen3 endpoint connects and then produces nothing.

This adds Qwen3STT and Qwen3TTS as separate classes. The existing Orpheus/Whisper paths are untouched, so this is non-breaking.

	`STT` / `TTS`	`Qwen3STT` / `Qwen3TTS`
STT audio	raw binary PCM	base64 `input_audio_buffer.append`
STT results	`message_type` / `transcript`	`type: "transcription"` / `segments[].text`
TTS transport	`{prompt, voice, …}`, or WS + `__END__` sentinel	`session.config` → `input.text` → `input.done`
TTS voices	preset names (`tara`)	registered voice clones

session = AgentSession(
    stt=baseten.Qwen3STT(model_id="your-qwen3-asr-model-id"),
    tts=baseten.Qwen3TTS(model_id="your-qwen3-tts-model-id", voice="your-voice"),
)

Both accept model_endpoint, model_id, or chain_id with the same precedence as STT (extracted into _endpoint.py).

Notes on the design

A few protocol details drove decisions that aren't obvious from the diff:

input.done is a flush, not a close. The session config stays in effect, so Qwen3TTS keeps one warm socket across turns. Re-dialing per utterance would add a connect plus a config round trip to every agent response.
Parked sockets need an application-level keepalive. The server has a 30s idle timeout that protocol pings don't reset, so an idle socket reads as OPEN long after the server has given up. An empty input.done answers session.done with zero sentences and proves the session is alive.
Interrupted sockets are discarded, not parked. Closing the socket is what stops in-flight generation; a graceful session.close would keep the GPU busy producing audio nobody hears.
One emitter segment per SynthesizeStream. push_text() after a flush is dropped by the framework and _main_task raises on a segment-count mismatch, so a mid-stream flush means "synthesize what's buffered", never "start a new segment".
Qwen3-ASR reports a language name ("English"), so Qwen3STT maps the common ones to ISO codes rather than passing a name where a code is expected.

Voices

Qwen3-TTS Base ships no built-in speakers — there's no tara equivalent. voice names a registered clone, and register_voice/list_voices are exported to manage them. Worth knowing: the server stores uploaded voices on the container's local disk, so a runtime-registered voice lives on one replica and is lost on restart. The README documents baking the reference into the deployment instead, or passing ref_audio/ref_text to clone inline per session.

Also: `language_options` for the existing Whisper `STT`

Bundled here because it is the same plugin and came out of the same customer
conversation. Baseten's streaming transcription API has accepted a
language_options list since Whisper runtime v0.5.0, which scopes detection to
the languages an agent actually supports. The plugin only ever sent a single
audio_language, forcing a choice between a fixed tag that mistranscribes the
other language and auto, which detects across all 99 and is unreliable on the
one- to two-second utterances typical of telephony.

stt = baseten.STT(model_id="...", language="auto", language_options=["en", "de"])

Only added to the handshake when non-empty — StreamingWhisperInput uses
extra="forbid", so sending it unconditionally would break anyone on an older
runtime. Also wired through update_options on both STT and SpeechStream.

Verified against a live Whisper Large V3 Turbo streaming deployment, with a
negative control: language_options: ["en", "de"] is accepted and transcribes
normally, while a deliberately misspelled field name closes the socket with
1011 — so acceptance confirms the field name rather than showing it was
silently ignored.

Testing

Developed against livekit-agents 1.6.8 with a mock server implementing the Qwen3 protocols, driven through the real framework machinery (AudioEmitter, RecognizeStream, the retry loop) and through a real AgentSession with a real-time audio sink. Covered:

TTS: token-by-token push_text, socket reuse without config resend, keepalive on a parked socket, barge-in discarding the socket, transient error retried / persistent error propagating, word timestamps rebased across sentence boundaries
STT: handshake shape, start_of_speech → interim → final → end_of_speech, one-shot recognize(), partials disabled via interim_results=False
AgentSession: TTS audio out with reuse across turns and interrupt() mid-playout; STT mic audio in surfacing interim + final user turns over consecutive VAD-bounded turns

I'm happy to contribute those as pytest suites if you'd like them in-tree — I left them out to keep the diff focused and avoid adding scripts your CI would try to collect.

Live validation

Since opening this, both adapters have been run end to end against real Baseten
deployments of the same model-registry trusses they target (Qwen3-ASR streaming
on RTX Pro 6000, Qwen3-TTS Base on RTX Pro 6000), using real speech with ground
truth rather than synthetic audio.

STT — a mu-bench en-US utterance, streamed at 100ms frames:

partial: I want to get a higher limit on my
FINAL:   Hi, I want to get a higher limit on my credit card. | lang: en
truth:   Hi. I want to get a higher limit on my credit card.

100% word overlap, both through Qwen3STT.stream() directly and through a real
AgentSession (user_input_transcribed, 6 interims + 1 final). Confirms the
handshake, the base64 append frames, type: "transcription" parsing, the
language_code: "English" -> en mapping, and clean termination on the
commit-triggered final.

TTS — voice.list returns {"voices": [], ...} on the Base checkpoint,
confirming it ships no built-in speakers; voice.add cloning from a 14s
reference works; synthesis returns real 24kHz PCM; the second turn reuses the
warm socket (TTFA 1081ms vs a cold first turn).

Round trip — feeding the live TTS output back into the live STT transcribes
at 100% word overlap, so the synthesized audio is genuinely intelligible speech
and not just well-formed bytes.

One operational note worth stating: on a cold replica the first synthesis
exceeded the 60s session timeout and was retried by the framework before
succeeding. That is cold-start behavior rather than an adapter issue, but
production voice agents should keep min_replica >= 1.

ruff check and ruff format are clean.

Baseten hosts Qwen3-ASR Streaming and Qwen3-TTS alongside the Whisper and Orpheus models this plugin already supports, but they speak different wire protocols, so the existing STT/TTS classes cannot reach them: - STT sends raw binary PCM and reads `message_type`/`transcript`; Qwen3-ASR takes base64 audio in OpenAI-realtime `input_audio_buffer.append` frames and replies with `type: "transcription"` / `segments[].text`. - TTS posts `{prompt, voice, ...}` (or a WS init frame plus an `__END__` sentinel); Qwen3-TTS uses `session.config` -> `input.text` -> `input.done`, where `input.done` is a flush rather than a close. Adds `Qwen3STT` and `Qwen3TTS` as separate classes so the existing Orpheus and Whisper paths are untouched. Both accept `model_endpoint`, `model_id`, or `chain_id` with the same precedence as `STT`. Qwen3TTS keeps one warm WebSocket across turns, since the session config is sticky and re-dialing would add a connect plus a config round trip to every agent response. Parked sockets are kept off the server's 30s idle timeout with an empty `input.done` flush (a protocol ping proves the socket is alive, not the session). Interrupted sockets are discarded rather than parked, because closing the socket is what stops in-flight generation. Qwen3-TTS Base ships no built-in speakers, so `voice` names a registered clone; `register_voice`/`list_voices` helpers are exported for managing them. Both support optional word-level timestamps via `TimedString`.

CLAassistant · 2026-08-04T22:25:21Z

All committers have signed the CLA.

- Import TimedString directly instead of guarding it behind a try/except. The fallback assigned None to a name mypy treats as a type, and in-tree the dependency is always current — every other plugin imports it unconditionally. - Drop the hand-rolled language-name table in favor of LanguageCode, which already normalizes names to codes ("English" -> "en"). Only Cantonese and Filipino are left untouched by it, so those stay as explicit overrides (Cantonese in particular is a distinct Qwen3-ASR language, not a zh variant). - Annotate the json.loads/dict.get returns in the voice-management helpers.

- Qwen3STT sent `input_audio_buffer.commit` twice at end of input. `end_input()` pushes a flush sentinel *and then* closes the channel, so the sentinel branch committed and the trailing block committed again with no audio in between, making the server answer a spurious empty turn. Track whether the last action was a commit, the way the TTS side already does. Covered by a regression test (verified failing before the fix). - Use BASETEN_MODEL_ENDPOINT in Qwen3STT rather than inventing BASETEN_STT_ENDPOINT. STT, TTS and Qwen3TTS all read the documented variable, so only Qwen3STT diverged — and resolve_endpoint claims to mirror STT. - Warn when an endpoint is plaintext ws:// to a non-loopback host. The API key travels in an Authorization header and the audio is unencrypted, so this is worth flagging; ws:// stays permitted for local proxies and tests. - Add Google-style docstrings to the new public classes and constructors, per CONTRIBUTING (pdoc3 generates the API reference from them).

Qwen3TTS._keepalive_loop held _warm_lock while awaiting _empty_flush, which sends input.done and waits for session.done. _acquire() needs the same lock on the hot path for every turn, so a reply that began mid-keepalive stalled for the rest of that round trip — measured at 1.2s against a deliberately slow server, on a warm pool whose whole purpose is to cut startup latency. Take the socket out of the slot before flushing and re-park it after. A turn arriving during the flush now finds an empty slot and dials its own socket instead of blocking, and the invariant that a turn and the keepalive never share a socket is preserved by construction rather than by the lock. If a turn parked its own socket meanwhile, the keepalive's is closed as surplus.

Baseten's streaming transcription API has accepted a `language_options` list since Whisper runtime v0.5.0, letting detection be scoped to the languages an agent actually supports. The plugin only ever sent a single `audio_language`, so users had to choose between a fixed tag that mistranscribes the other language and `auto`, which detects across all 99 and is unreliable on the one- to two-second utterances typical of telephony. The field is only added to the handshake when non-empty: `StreamingWhisperInput` uses `extra="forbid"`, so unconditionally sending it would break anyone on a runtime older than v0.5.0. Verified against a live Whisper Large V3 Turbo streaming deployment: the handshake with `language_options: ["en", "de"]` is accepted and transcribes normally, while a deliberately misspelled field name closes the socket with 1011 — confirming both the name and that rejection is real rather than silent.

- Replace the hand-rolled Qwen3ChunkedStream with the framework's `_synthesize_with_stream()`. Mine passed the caller's conn_options straight to the inner stream, so a persistent failure retried (max_retry+1)^2 times — nine attempts at the default — and it never forwarded USERDATA_TIMED_TRANSCRIPT, silently dropping the word timings the class advertises via aligned_transcript. Four other streaming-only plugins already use this helper. - Close the keepalive's in-flight socket when the task is cancelled. The previous commit deliberately removed the socket from the warm slot before flushing (to keep the lock off the hot path), which meant aclose() during a flush cancelled the only reference and leaked the connection. - Anchor Qwen3STT timings to the session clock. Segment and word times were reported socket-relative, so after a reconnect they jumped back toward zero. The framework grows `start_time_offset` for exactly this, and the existing Whisper STT already applies it. Regression tests added for the timing offset and for one-shot synthesis (asserts a single attempt and that timed transcripts survive). Not changed: `resolve_endpoint` still warns rather than rejects plaintext ws:// to non-loopback hosts. Hard-failing would break local proxy and test setups, and the existing TTS/STT in this plugin accept ws:// with no check at all, so the warning is already stricter than the status quo.

…plies `_control` parsed `(await ws.receive()).data` as JSON without inspecting the frame type. aiohttp gives `.data` as None on CLOSED, the close code on CLOSE, and an exception on ERROR, so a deployment that drops the connection after the auth handshake produced a bare TypeError instead of the tailored errors `register_voice`/`list_voices` raise. The rest of this module already switches on `msg.type` before parsing; this brings the helpers in line. A dropped connection now reports: unexpected reply to 'voice.list': CLOSE Still not changed: plaintext ws:// to a non-loopback host warns rather than raises. Rejecting it outright would break local proxy and test setups, and the pre-existing TTS/STT in this plugin accept ws:// with no check at all.

devin-ai-integration

Devin Review found 1 new potential issue.

View 3 additional findings in Devin Review.

devin-ai-integration · 2026-08-05T01:02:38Z

+            async with self._warm_lock:
+                surplus = self._warm is not None
+                if not surplus:
+                    self._warm = warm
+            if surplus:
+                # A turn started during the flush and parked its own socket on
+                # the way out, so ours is now redundant.
+                await self._shutdown_ws(warm.ws, notify=True)
+                return


🟡 Kept-alive voice connection can silently stop being refreshed, forcing a slow reconnect on the next reply

The background refresher for the parked voice connection gives up permanently (return at livekit-plugins/livekit-plugins-baseten/livekit/plugins/baseten/qwen3_tts.py:435) when a reply happens to park a newer connection while it is mid-check, and nothing restarts it until the reply after that, so the parked connection quietly goes stale.
Impact: An idle agent can lose its warm connection, adding a full reconnect delay before the user hears the next spoken response.

Race between `_keepalive_loop` exiting and `_release` skipping the restart

_keepalive_loop removes the parked socket from self._warm (qwen3_tts.py:413) before doing the round trip in _empty_flush. A turn that starts during that window finds an empty slot, dials its own socket, and on completion _release parks it. Because the old keepalive task is still running (inside _empty_flush, up to _KEEPALIVE_TIMEOUT), the guard at qwen3_tts.py:386 (self._keepalive_task is None or self._keepalive_task.done()) is false, so no new keepalive is started. The old loop then observes surplus and returns at qwen3_tts.py:431-435, leaving the freshly parked socket with no keepalive at all.

The server drops idle sessions after 30s; _CONFIGURED_TTL (25s) means _acquire will correctly discard the stale socket rather than use a dead one, so this is a lost optimization rather than a failure — but the warm-socket path this class exists for is silently disabled until the next _release observes self._keepalive_task.done().

Prompt for agents

In livekit-plugins/livekit-plugins-baseten/livekit/plugins/baseten/qwen3_tts.py, `_keepalive_loop` can terminate while a live socket is parked in `self._warm`, and `_release` will not restart it because the (still-running) task was not `done()` at the time it parked the socket. Sequence: keepalive pulls the socket out of `self._warm` and starts `_empty_flush`; concurrently a turn acquires (empty slot), dials, finishes, and calls `_release`, which parks a new socket but skips starting a keepalive since the old task is still running; the old keepalive then sees `surplus` and returns. The newly parked socket now has no keepalive and will idle out on the server. Possible approaches: instead of returning in the surplus branch, discard the redundant socket and continue looping (the loop already re-reads `self._warm` each iteration); or have `_release` unconditionally ensure a keepalive is scheduled (e.g. an explicit `_ensure_keepalive()` that the loop itself calls on exit while a socket is parked). Make sure `aclose()`/`_closing` still terminates the loop and that no socket is left unclosed.

Was this helpful? React with 👍 or 👎 to provide feedback.

iancarrasco-b10 · 2026-08-05T01:02:47Z

+from .log import logger
+
+_TRUSS_URL_TEMPLATE = "wss://model-{model_id}.api.baseten.co/environments/production/websocket"
+_LOOPBACK = {"localhost", "127.0.0.1", "::1"}


ooc what is reason for this

It's the exemption list for the plaintext-ws:// warning further down in resolve_endpoint:

if endpoint.startswith("ws://") and urlparse(endpoint).hostname not in _LOOPBACK: logger.warning("endpoint %r is plaintext ws://: the Baseten API key and all audio " "will be sent unencrypted. Use wss:// for any non-local host.", endpoint)

We send Authorization: Api-Key ... on that connection, so a ws:// endpoint to a remote host puts the key and the audio in cleartext. Loopback is exempt because local proxies and the tests legitimately use ws://127.0.0.1 and nothing leaves the machine.

Warning rather than rejecting was deliberate — hard-failing would break those local setups, and the existing TTS/STT in this plugin accept ws:// with no check at all today.

Fair callout though: it was wedged between the two URL templates with no context, which is exactly why it read as arbitrary. Moved it next to the logic it serves and added a comment in 843eac0.

Answers review feedback: _LOOPBACK sat between the two URL templates with no context, reading as arbitrary. Group it with the warning it serves and say what it is for — hosts exempt from the plaintext-ws:// warning, because local proxies and tests legitimately use ws://127.0.0.1.

- Feed the TTS stall watchdog from incoming audio. `_SESSION_DONE_TIMEOUT` was only refreshed by the sender finishing and by `session.done`, but the server sends `session.done` once the *whole* utterance is synthesized — so a reply taking longer than 60s was aborted mid-sentence on a healthy socket actively delivering PCM, and because audio had already been pushed the framework refuses to retry. Setting progress on each binary frame keeps the timeout a genuine idle watchdog. (This is what I saw on a cold replica during live testing and wrongly wrote off as cold-start noise.) - Only open an STT turn on actual words. An empty final — Silero closing a turn on noise with nothing recognized — emitted START_OF_SPEECH and END_OF_SPEECH with no transcript, and under turn_detection="stt" that commits the user's turn and makes the agent answer silence. Both covered by regression tests verified failing beforehand: the long-turn one raised APITimeoutError, the empty-turn one emitted a spurious start_of_speech/end_of_speech pair.

A-K-Erol requested a review from a team as a code owner August 4, 2026 22:25

This comment was marked as resolved.

Sign in to view

A-K-Erol added 2 commits August 4, 2026 15:40

This comment was marked as resolved.

Sign in to view

A-K-Erol changed the title ~~feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support~~ feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support, and language_options for Whisper STT Aug 5, 2026

Navigation Menu

Uh oh!

feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support, and language_options for Whisper STT - #6700

feat(baseten): add Qwen3-ASR STT and Qwen3-TTS support, and language_options for Whisper STT#6700
A-K-Erol wants to merge 9 commits into
livekit:mainfrom
A-K-Erol:baseten-qwen3-tts-stt

A-K-Erol commented Aug 4, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Aug 4, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Aug 5, 2026

Uh oh!

iancarrasco-b10 Aug 5, 2026

Uh oh!

A-K-Erol Aug 5, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

A-K-Erol commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes on the design

Voices

Also: language_options for the existing Whisper STT

Testing

Live validation

Uh oh!

CLAassistant commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

iancarrasco-b10 Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

A-K-Erol Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

A-K-Erol commented Aug 4, 2026 •

edited

Loading

Also: `language_options` for the existing Whisper `STT`

CLAassistant commented Aug 4, 2026 •

edited

Loading