Don't lex '< text' (whitespace after '<') as html_element (bd-ly83qewg) - #469
Merged
Merged
Conversation
The external scanner's parse_open_angle_brace walked from any inline
'<' to the next '>' and emitted a best-effort HTML_ELEMENT token, even
when whitespace immediately followed the '<'. A construct like
*a < b text.* a > b
lexed '< b text.* a >' as one html_element, swallowing the emphasis
closer and failing with Q-2-12 Unclosed Star Emphasis — while deleting
the '>' made the error vanish (the bd-j9cf EOF fallback kicked in).
Per HTML/CommonMark, a tag name (or '/', '!', '?') must immediately
follow '<', and autolinks admit no whitespace, so whitespace or EOF
right after '<' disqualifies every angle-bracket construct except the
raw specifier ('{<reader}'). The fix:
- guard the HTML_ELEMENT arm with html_possible (no whitespace/EOF
immediately after '<');
- fast path: when html_possible is false and RAW_SPECIFIER is not a
valid symbol, emit LT_STR_LITERAL immediately instead of scanning to
the next '>'/EOF (also avoids an O(n) walk per '< ' occurrence);
- the RAW_SPECIFIER scan path is byte-for-byte unchanged.
Whitespace before '>' deliberately does NOT disqualify: '<div >' is a
valid open tag. Interior-only whitespace ('<not a tag>') also still
lexes as html_element; that ambiguity is out of scope.
Tests (written first, verified failing): 3 new corpus cases + 2
regression pins in lt-as-str.txt; 3 behavior tests + 2 RawInline
guards in pampa's test_bare_lt_str.rs. Full tree-sitter corpus (545)
and cargo xtask verify green. No snapshot (.snap) changes.
Output now matches pandoc -t native for the repro (emphasis closes,
'<' and '>' are literal Strs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The external scanner's
parse_open_angle_bracewalked from any inline<to the next>and emitted a best-effortHTML_ELEMENTtoken, even when whitespace immediately followed the<. A construct likelexed
< b text.* a >as onehtml_element, swallowing the emphasis closer and failing with Q-2-12 Unclosed Star Emphasis — while deleting the>made the error vanish (the bd-j9cf EOF fallback then emittedLT_STR_LITERALinstead).Fix
Per HTML/CommonMark, a tag name (or
/!?) must immediately follow<, and autolinks admit no whitespace, so whitespace or EOF right after<disqualifies every angle-bracket construct except the raw specifier ({<reader}):HTML_ELEMENTarm withhtml_possible(no whitespace/EOF immediately after<);RAW_SPECIFIERisn't a valid symbol, emitLT_STR_LITERALimmediately instead of scanning to the next>/EOF (avoids an O(n) walk per<occurrence);RAW_SPECIFIERscan path byte-for-byte unchanged.Deliberate non-changes: whitespace before
>does not disqualify (<div >is a valid open tag), and the scan still crosses newlines (<div\n class="foo">is valid); interior-only whitespace (<not a tag>) still lexes ashtml_element— that ambiguity is out of scope.Output now matches
pandoc -t nativefor the repro (emphasis closes;<and>are literalStrs).Tests
Written first and verified failing (repro died with Q-2-12):
test/corpus/lt-as-str.txtRawInlineregression guards in pampa'stest_bare_lt_str.rsVerification: tree-sitter corpus 545/545; full
cargo xtask verifygreen (workspace build + tests, WASM leg, hub-client build + tests); no.snapchanges.Plan:
claude-notes/plans/2026-08-07-angle-bracket-inner-whitespace.md(bd-ly83qewg)🤖 Generated with Claude Code