fix(tokenize): don't stall TTS streaming on angle-bracketed prose

[priyam-garg]

_XML_TAG_RE matched any <letter...>, so a markdown autolink or an angle-bracketed
address parsed as an open XML tag. With xml_aware=True (expressive agents) the depth
counter in _has_unclosed_xml_tags then never returned to zero, and every later
sentence was held until flush — streaming TTS degraded to end-of-turn batching for the
rest of the turn.

Measured on main, first sentence emitted before flush:

text                                          xml_aware=True   xml_aware=False
Docs are at <https://docs.livekit.io> now.    stalled          streams
Email me at <bob@example.com> please.         stalled          streams
Press <ctrl+c> to quit.                       stalled          streams
Note that 3 < 5 holds.                        streams          streams

Require the tag name to be followed by whitespace, / or >, as it is in real markup.
Hyphens join the name so xAI's <higher-pitch> / <build-intensity> still match.

_XML_TAG_RE = re.compile(r"<(/?)([A-Za-z][\w-]*)(?=[\s/>])[^>]*?(/?)\s*>")

This is the same regression class as the bare < and digit-named <5> fixes that
TestPlainTextAngleBrackets already documents; the existing
test_tag_shaped_text_streams_when_not_xml_aware covers <bob@example.com> but only
asserts the xml_aware=False path, so the broken case sat next to a passing test.

filter_markdown does not remove these spans (it rewrites [text](url) but leaves
autolinks and addresses intact), so they do reach the tokenizer.

All three consumers of the regex were checked — the depth counter, _is_xml_only, and
_xml_wrap_tokenizer's strip-and-remap. Stricter matching is correct for all three: an
autolink is text, so it shouldn't be stripped before blingfire either.

Testing

Full --unit gate, before and after:

	passed	skipped	errors
main (baseline)	1335	5	9
this branch	1339	5	9

The +4 are the new tests. The 9 errors are a pre-existing Event loop is closed
INTERNALERROR from tests/concurrency.py on Python 3.14 — identical on both sides,
unrelated to this change.

I don't have data on how often models emit angle-bracket autolinks, so I'm not claiming
a frequency. The case for the fix is that it's one line, strictly more correct, and when
it does fire the cost is every remaining sentence in the turn.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[Copilot]

Pull request overview

This PR fixes an XML-markup detection regression in the sentence/token streaming layer that caused expressive (xml_aware=True) TTS streaming to stall after encountering angle-bracketed prose (e.g., markdown autolinks, emails, key chords). It tightens the XML tag-name matcher so only markup-shaped constructs affect the “unclosed tag depth” logic, restoring sentence-by-sentence streaming.

Changes:

• Refines _XML_TAG_RE to require the tag name be followed by whitespace, /, or > (and allows hyphenated tag names).
• Adds targeted unit tests for angle-bracketed prose and hyphenated tags to prevent depth-counter stalls.
• Adds end-to-end streaming tests ensuring autolinks/emails still stream with xml_aware=True.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
livekit-agents/livekit/agents/tokenize/token_stream.py	Tightens XML tag regex so angle-bracketed prose doesn’t increment XML depth and stall streaming, while keeping real (including hyphenated) tags working.
tests/test_tokenizer_xml_markup.py	Adds regression + end-to-end streaming coverage for autolinks/emails under xml_aware=True and ensures hyphenated tags remain recognized.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.