Skip to content

Don't lex '< text' (whitespace after '<') as html_element (bd-ly83qewg) - #469

Merged
cscheid merged 1 commit into
mainfrom
bugfix/bd-ly83qewg-angle-inner-whitespace
Aug 8, 2026
Merged

Don't lex '< text' (whitespace after '<') as html_element (bd-ly83qewg)#469
cscheid merged 1 commit into
mainfrom
bugfix/bd-ly83qewg-angle-inner-whitespace

Conversation

@cscheid

@cscheid cscheid commented Aug 8, 2026

Copy link
Copy Markdown
Member

Summary

The external scanner's parse_open_angle_brace walked from any inline < to the next > and emitted a best-effort HTML_ELEMENT token, even when whitespace immediately followed the <. A construct like

*a < b this text is interpreted as an HTML element.* a > b

lexed < b text.* a > as one html_element, swallowing the emphasis closer and failing with Q-2-12 Unclosed Star Emphasis — while deleting the > made the error vanish (the bd-j9cf EOF fallback then emitted LT_STR_LITERAL instead).

Fix

Per HTML/CommonMark, a tag name (or / ! ?) must immediately follow <, and autolinks admit no whitespace, so whitespace or EOF right after < disqualifies every angle-bracket construct except the raw specifier ({<reader}):

  • guard the HTML_ELEMENT arm with html_possible (no whitespace/EOF immediately after <);
  • fast path: when HTML is impossible and RAW_SPECIFIER isn't a valid symbol, emit LT_STR_LITERAL immediately instead of scanning to the next >/EOF (avoids an O(n) walk per < occurrence);
  • RAW_SPECIFIER scan path byte-for-byte unchanged.

Deliberate non-changes: whitespace before > does not disqualify (<div > is a valid open tag), and the scan still crosses newlines (<div\n class="foo"> is valid); interior-only whitespace (<not a tag>) still lexes as html_element — that ambiguity is out of scope.

Output now matches pandoc -t native for the repro (emphasis closes; < and > are literal Strs).

Tests

Written first and verified failing (repro died with Q-2-12):

  • 3 new corpus cases + 2 regression pins in test/corpus/lt-as-str.txt
  • 3 behavior tests + 2 RawInline regression guards in pampa's test_bare_lt_str.rs

Verification: tree-sitter corpus 545/545; full cargo xtask verify green (workspace build + tests, WASM leg, hub-client build + tests); no .snap changes.

Plan: claude-notes/plans/2026-08-07-angle-bracket-inner-whitespace.md (bd-ly83qewg)

🤖 Generated with Claude Code

The external scanner's parse_open_angle_brace walked from any inline
'<' to the next '>' and emitted a best-effort HTML_ELEMENT token, even
when whitespace immediately followed the '<'. A construct like

    *a < b text.* a > b

lexed '< b text.* a >' as one html_element, swallowing the emphasis
closer and failing with Q-2-12 Unclosed Star Emphasis — while deleting
the '>' made the error vanish (the bd-j9cf EOF fallback kicked in).

Per HTML/CommonMark, a tag name (or '/', '!', '?') must immediately
follow '<', and autolinks admit no whitespace, so whitespace or EOF
right after '<' disqualifies every angle-bracket construct except the
raw specifier ('{<reader}'). The fix:

- guard the HTML_ELEMENT arm with html_possible (no whitespace/EOF
  immediately after '<');
- fast path: when html_possible is false and RAW_SPECIFIER is not a
  valid symbol, emit LT_STR_LITERAL immediately instead of scanning to
  the next '>'/EOF (also avoids an O(n) walk per '< ' occurrence);
- the RAW_SPECIFIER scan path is byte-for-byte unchanged.

Whitespace before '>' deliberately does NOT disqualify: '<div >' is a
valid open tag. Interior-only whitespace ('<not a tag>') also still
lexes as html_element; that ambiguity is out of scope.

Tests (written first, verified failing): 3 new corpus cases + 2
regression pins in lt-as-str.txt; 3 behavior tests + 2 RawInline
guards in pampa's test_bare_lt_str.rs. Full tree-sitter corpus (545)
and cargo xtask verify green. No snapshot (.snap) changes.

Output now matches pandoc -t native for the repro (emphasis closes,
'<' and '>' are literal Strs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@posit-snyk-bot

posit-snyk-bot commented Aug 8, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
cscheid merged commit 0e2d889 into main Aug 8, 2026
8 checks passed
@cscheid
cscheid deleted the bugfix/bd-ly83qewg-angle-inner-whitespace branch August 8, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants