Apply LB30 East Asian exception for opening punctuation - #53
Open
dngr2 wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the Unicode line-breaking data generation and pair table so Rule LB30’s East Asian exception is applied to wide/fullwidth/halfwidth opening punctuation (and related closing parens), fixing previously incorrect “no-break” behavior before wide brackets and allowing more LineBreakTest.txt rows to pass.
Changes:
- Extend
generate_data.jsto fetch and parseEastAsianWidth.txt, splitting OP/CP intoOP_EA/CP_EAduring trie generation. - Expand the pair table to include the new East Asian-specific break classes.
- Reduce the
LineBreakTest.txtskip list now that more cases pass.
Reviewed changes
Copilot reviewed 3 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/index.js | Removes now-passing test rows from the skip list. |
| src/pairs.js | Adds OP_EA / CP_EA columns/rows to the pair table for LB30 East Asian handling. |
| src/generate_data.js | Fetches/parses EastAsianWidth.txt and assigns OP_EA / CP_EA during trie generation. |
| src/classes.js | Introduces new break-class constants for East Asian OP/CP and renumbers following classes. |
| src/classes-trie-data.js | Updates generated trie data to reflect the new class assignments. |
Files not reviewed (2)
- src/classes-trie-data.js: Generated file
- src/pairs.js: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| [IN_BRK, PR_BRK, PR_BRK, IN_BRK, IN_BRK, IN_BRK, PR_BRK, PR_BRK, PR_BRK, IN_BRK, IN_BRK, IN_BRK, IN_BRK, IN_BRK, DI_BRK, IN_BRK, IN_BRK, IN_BRK, DI_BRK, DI_BRK, PR_BRK, CI_BRK, PR_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, IN_BRK, DI_BRK, IN_BRK, PR_BRK], // ZWJ | ||
| [DI_BRK, PR_BRK, PR_BRK, IN_BRK, IN_BRK, DI_BRK, PR_BRK, PR_BRK, PR_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, PR_BRK, CI_BRK, PR_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, IN_BRK, DI_BRK, DI_BRK, PR_BRK], // CB | ||
| [PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, CP_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK], // OP_EA | ||
| [DI_BRK, PR_BRK, PR_BRK, IN_BRK, IN_BRK, PR_BRK, PR_BRK, PR_BRK, PR_BRK, IN_BRK, IN_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, IN_BRK, IN_BRK, IN_BRK, DI_BRK, DI_BRK, PR_BRK, CI_BRK, PR_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, DI_BRK, IN_BRK, DI_BRK, DI_BRK, PR_BRK], // CP_EA |
Comment on lines
+54
to
+61
| request(`${BASE}EastAsianWidth.txt`, function (eawErr, eawRes, eawData) { | ||
| const isEastAsianWide = parseEastAsianWide(eawData); | ||
|
|
||
| // this loads the LineBreak.txt file for Unicode and parses it to | ||
| // combine ranges and generate JavaScript | ||
| request(`${BASE}LineBreak.txt`, function (err, res, data) { | ||
| const matches = data.match(/^[0-9A-F]+(\.\.[0-9A-F]+)?;[A-Z][A-Z0-9]([A-Z])?/gm); | ||
|
|
Rule LB30 prohibits a break before opening punctuation only when the bracket is not East Asian wide/full/halfwidth, but all OP was treated the same, so a break before a wide bracket such as U+2329 or U+FF08 was wrongly prohibited. generate_data.js now also reads EastAsianWidth.txt and puts East Asian opening punctuation in its own class (OP_EA) so LB30's exception applies. Un-skips the 15 LineBreakTest.txt rows this fixes.
dngr2
force-pushed
the
lb30-east-asian-opening-punctuation
branch
from
August 17, 2026 01:56
17fafcc to
7c6c20a
Compare
Author
|
Addressed the review: dropped the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rule LB30 only prohibits a break before opening punctuation when the bracket is not East Asian (wide / fullwidth / halfwidth). The library treated all
OPthe same, so it wrongly prohibited breaks before wide brackets:generate_data.jsnow also readsEastAsianWidth.txtand puts East Asian opening punctuation / closing parens in their own break classes (OP_EA/CP_EA), so LB30's exception applies. This makes 15 of the previously-skippedLineBreakTest.txtrows pass — I removed those from the skip list. The remaining skips are the numeric (LB25) tailoring cases, which are unaffected.npm test: 7610 passing, 44 pending.