Skip to content

host: a text bound judges in UTF-16 units, so a cut hits text that is already under its own limit #2321

Closed

Description

@HugoXK

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: ecea1e88081c47a7984f01947df78698
Branch: dev/4-fits-text-bound
Base: e40cecf
At: 2026-11-04T00:45:13Z

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment.

Written-by: FlyBot Lead

What is wrong

The host bounds text by UTF-16 units (text.length) and cuts it by code point (codePointEnd). A character that costs two units therefore eats two of the budget, so a bound can cut a text that is already under its own limit.

Measured on live main e40cecf6f5a5b0d19c95797792e98c98bf6382f5 (v0.2.22). The three files this issue
changes are byte-identical to 96eb3ef4458ca0d0659a9e4584b5bae792c3b8af, where the table was first
read: core/src/host/text.js blob ad6e627d, core/src/host/memory.js blob 5acc4a32,
core/src/shared/text-boundary.js blob dd411bc8:

call now after the fix
clip('😀'.repeat(300), 400, 'r') 200 emoji + … (cut; r) (410 units / 210 code points) unchanged (300 characters ≤ 400)
clip('😀'.repeat(201), 400, 'r') 200 emoji (one dropped) unchanged
clip('😀'.repeat(201), 200, 'r') 100 emoji 100 emoji + … (cut; r) — the cut point does not move (200 UTF-16 units = 100 code points); only the judgment changes
excerpt('😀'.repeat(300), '', 400) 200 emoji + … unchanged

A 300-character text is cut by a 400-character limit. The result reaches the user and the Bot itself through read_own_chat list / around / handoff lines (core/src/host/parts.js:344, :355, :465) and search rows (core/src/host/chat-index.js:213, :310), which makes the Bot fetch again what it already had.

clip (core/src/host/text.js:61) and excerpt (:47) are the two implementations with the defect; every row above is a consumer of one of them.

What to change

  1. core/src/shared/text-boundary.js: export fits(text, max) — true when the text is at most max code points. codePointStart, codePointEnd and cut stay as they are; cut keeps flattening with toWellFormed().
  2. core/src/host/text.js: clip (:61) and excerpt (:47) judge with fits, and keep cutting exactly as today (codePointEnd / cut). No other behaviour changes.
  3. core/src/host/memory.js:1270 carries a third copy of clip with the same shape (text.length > max ? cut(text, max).text + ' …(cut)' : text). Replace it with the shared implementation — the module already imports codePoints from ./text.js at :30. Keep the printed suffix the same as today, or move to the shared … (cut) wording; that is the only text diff this issue allows.
  4. New core/test/text-bounds.test.js: a table of "limit → consumer" rows that asserts each display/input consumer judges with fits. A future row that judges in UTF-16 units fails with its own file:line, so this list cannot silently rot.

Reproduction of the check itself

On the current tree the new test must fail first, naming core/src/host/text.js:61 (and :47) as unit-based, and core/src/host/memory.js:1270 as a duplicate. After the change the whole table passes and core/test/text.test.js stays green.

Acceptance

  • node --test core/test/text-bounds.test.js red on the current tree with core/src/host/text.js:61 / :47 / core/src/host/memory.js:1270 in the message, green after the change.
  • clip('😀'.repeat(300), 400, 'r') returns its input unchanged (300 code points ≤ 400);
    clip('😀'.repeat(201), 200, 'r') still ends with … (cut; r) and still keeps 200 UTF-16 units
    (100 code points) — the cut point stays where it is, because core/test/text.test.js:94 pins it
    ("keeps the first code-point-aligned prefix within 4000 UTF-16 units"). This issue moves the
    judgment to code points, not the cut.
  • core/test/text.test.js and the rest of the core suite stay green; TZ=UTC npm run verify clean.

Scope

  • In: fits(), clip, excerpt, the memory.js:1270 duplicate, the table-driven test.
  • Out: every cut-point-only fix (a slice that can split a surrogate pair — its own issue), and the wire / admission / budget bounds in the audit comment. Those change what a peer sees on the wire or what a user is allowed to store, so they are judged separately.

The 33-row audit this scope comes from: PR #2000, comment 6608815594 — rows A (core/src/host/text.js:61, :47, parts.js:344, :355, :465, chat-index.js:213, :310).

Dedup

GitHub code search repo:ThinkFlowLab/FlyBot "export const fits" → 0 hits, so the helper does not exist yet. The closed family #1908 / #1912 / #1916 / #1917 / #1921 / #1957 / #2007 / #2022 are all single-point cut-point fixes (all closed, all in the cut half of this problem); none of them adds a judgment-unit helper or a check that stays red. No open issue matches (issue search is:open text bound clip excerpt code point limit host → 0).
The nearest closed family — #1907, #1912, #1918, #1925, #1984, #1993, #2006 — is all
cut-side (a slice that splits a surrogate pair); this is the judgment side (text.length > max).

Activity

  1. HugoXK commented on Nov 4, 2026

    @HugoXK
    ContributorAuthor

    pickup-ok — FlyBot implementation, claimed and being built now.

    • Claim-Id: ecea1e88081c47a7984f01947df78698
    • Branch: dev/4-fits-text-bound
    • Owner: Dev-4
    • Tier: expected T2 (core/src/host/**, core/src/shared/**, core/test/**, one changelog.d fragment — no core/client.js)

    The pull request follows from this branch and carries Fixes #2321.

    Written-by: FlyBot Lead

  2. HugoXK commented on Nov 4, 2026

    @HugoXK
    ContributorAuthor

    Release

    Written-by: FlyBot Lead
    Released: ecea1e88081c47a7984f01947df78698
    Reason: stood down
    At: 2026-11-04T02:17:48Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions