feat(openai): support context hints in STT - #6705
Conversation
gpt-transcribe and gpt-live-transcribe accept `keywords` (literal terms expected in the audio) and a plural `languages` list for code-switched speech. `language` now takes a `str` or a `list[str]`, and raises when a list of more than one reaches a model that only accepts a single language. Codes are normalized to ISO-639-1, which the API requires: it rejects regional tags such as en-US. `keywords` also lets the plugin join the keyterm framework, so `stt_context_options` and auto keyterm detection now reach OpenAI STT. Detected terms merge behind the user's own and are applied with a `session.update` on the open connection, since the detector moves them every turn and a reconnect per turn would be costly. Only a new `model` reconnects, because gateways route on the `?model=` in the upgrade URL. gpt-live-transcribe rejects any turn_detection config and emits no speech_started/stopped, so it joins gpt-realtime-whisper on the client-commit path. Without that the model fails on connect and never produces a final transcript. Requires openai>=2.50, where `keywords`/`languages` became typed on both the file and realtime transcription APIs.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Configure the transcription session when the stream acquires a connection rather than when the pool opens one. The pool hands back sockets it opened earlier without calling the connect callback, so an option change made while no stream was live was lost for the reused socket. Refresh the transcript language tag on the reconnect path too, so changing model and language together no longer labels later transcripts with the old language. Drop detected keyterms from the effective set when the model does not accept keywords, so they can no longer make a later stream() or recognize() raise.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A field left out of a session.update keeps its previous value, so removing keywords or a prompt never reached an open connection. Both are now always sent, cleared as an empty array and an empty string. `languages` accepts neither an empty array nor null, so clearing it reconnects instead. Convert a failed configuration write into APIConnectionError. The write now lands on connections the pool may have opened much earlier, and one that died while idle would otherwise kill the stream instead of retrying. Reject a switch to a model without server-side endpointing when no VAD is available to commit the audio buffer, which used to leave the session with no final transcripts and no error. An explicit `vad=None` still opts out.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…setup The config a connection is set up with was built and sent while the socket was still unpublished, so a change arriving in that window was dropped as having no live connection and the session kept the old settings until its next rebuild. Publish the socket first so such a change is applied rather than discarded, and serialize the setup send against later updates. Publishing alone is not enough: a queued update could otherwise reach the socket before the setup config and be overwritten by it.
The client VAD kept committing the audio buffer after a switch away from gpt-live-transcribe, so server-side turn detection and the VAD both closed the same segment. The commit now only happens while the model has no endpointing of its own. use_realtime defaults to True for gpt-realtime-whisper and gpt-live-transcribe, which OpenAI serves only on the realtime endpoint, and both are now named in STTModels. The VAD those two need comes from the bundled inference.VAD rather than an optional livekit-plugins-silero import, so the ImportError path is gone. The model names behind these switches move into two constants, one per axis: the transport a model is served on, and whether it takes context hints.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| """ # noqa: E501 | ||
|
|
||
| if not is_given(use_realtime): | ||
| use_realtime = _requires_client_commit(model) |
There was a problem hiding this comment.
nit: we should warn if it is False while the model is streaming.
gpt-transcribe reports the languages it detected, on both the transcriptions endpoint and the realtime session. The transcript now carries the dominant one instead of the hint that was sent, which left code-switched audio untagged because a multi-language hint has no single code to report. update_options rejects a switch to a realtime-only model when the instance was built for the transcriptions endpoint, which does not serve those models at all. The transport cannot move instead: AgentSession wraps a non-streaming STT in a StreamAdapter once, so `streaming` cannot flip under a running pipeline. _requires_client_commit becomes _is_realtime_only, naming the model group rather than one of its consequences, the way _supports_context_hints mirrors _CONTEXT_HINT_MODELS.
| if model_changed or languages_cleared: | ||
| stream.reconnect() | ||
| else: | ||
| stream.apply_options() |
There was a problem hiding this comment.
🟡 Changing the transcription model or clearing the language leaves an idle reused connection configured with the old settings
Settings that can only be applied by rebuilding a connection are only rebuilt through currently open streams (stream.reconnect() at livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:534), so when no stream is open at that moment the previously opened connection stays in the pool and is later reused with the old settings.
Impact: After switching models or turning on automatic language detection between two speech sessions, transcription can keep using the previous model routing or the previous language, or fail outright.
Why an idle pooled socket keeps the stale configuration
STT.update_options computes model_changed / languages_cleared and relies on iterating self._streams to call SpeechStream.reconnect(), which is what performs self._pool.invalidate() (livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py:693-697). self._streams is a weakref.WeakSet, so after a stream is closed and dropped (a normal occurrence between agent activities) the set can be empty and nothing invalidates the pool.
The connection pool keeps the socket available for reuse (livekit-agents/livekit/agents/utils/connection_pool.py:95-122). When a new stream acquires it, _run re-sends the config built by _session_update, which is dumped with exclude_unset=True; fields that are absent keep their previous server-side value:
languagesis intentionally omitted when empty (stt.py:149-153), sodetect_language=Truenever clears the previously configured languages on that socket.turn_detectionis omitted forgpt-live-transcribe/gpt-realtime-whisper(stt.py:165-167), so a socket previously configured withserver_vadkeeps it while the model is switched to one that rejects turn detection.- The gateway routing model is fixed by the
?model=query parameter chosen at connect time (stt.py:551-559), so a model change cannot take effect on a reused socket at all.
A fix is to invalidate the pool directly in update_options whenever model_changed or languages_cleared, independently of whether any stream is currently registered.
Prompt for agents
In livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py, STT.update_options decides that a model change or a cleared language list requires rebuilding the websocket, but the only code path that actually drops the pooled connection is SpeechStream.reconnect(), which calls self._pool.invalidate(). That path is reached by iterating self._streams, a WeakSet that can be empty (all streams closed/collected) while the connection pool still holds an idle, already-configured socket for reuse. When a later stream acquires that socket, the setup session.update omits `languages` when empty and omits `turn_detection` for realtime-only models, so the previous values stick server-side, and the gateway routing model baked into the connect URL cannot change at all. Consider invalidating the pool from update_options itself whenever model_changed or languages_cleared, in addition to notifying any live streams.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
gpt-transcribeandgpt-live-transcribeaccept context hints:keywordsfor literal terms expected in the audio, and a plurallanguageslist for code-switched speech.languagenow takes astror alist[str]; more than one language raises on models that accept only one, and codes are normalized to ISO-639-1 because the API rejects regional tags such asen-US.keywordsgives the plugin somewhere to put terms, sostt_context_optionsand automatic keyterm detection now reach OpenAI STT. Detected terms merge behind the user's own and apply with asession.updateon the open connection; only a change ofmodelreconnects.gpt-live-transcriberejects anyturn_detectionconfig and emits nospeech_started/speech_stopped, so it joinsgpt-realtime-whisperon the client-commit path.openai>=2.50, wherekeywords/languagesbecame typed on both transcription APIs.