Skip to content

host: the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules) - #2000

Merged
Birfy merged 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/2-p2-c5-remaining
Nov 1, 2026
Merged

Birfy merged 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/2-p2-c5-remaining

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1993

Owner: Dev-2
Reviewer: Birfy
Written-by: Dev-2
Claim-Id: 9d717c402114410994aa3c6fcbec7118

What changed

Five bounds in core/src/host/ cut with a bare slice, so a bound that landed between the halves of a surrogate pair stored a lone high surrogate — read as U+FFFD by any encoder, and shown that way:

site what it bounds max before after
core/src/host/steward.js:148 the quoted first line of the kept tail in a checkpoint note 80 text.slice(0, 80) cut(text, 80).text
core/src/host/approvals.js:75 an approval record's reason 500 String(request.reason ?? '').slice(0, 500) cut(String(request.reason ?? ''), 500).text
core/src/host/hooks.js:1041 a mention receipt's reason line 240 …trim().slice(0, 240) cut(…trim(), 240).text
core/src/host/agents.js:169 the title given to a DSH Agent's Session AGENT_SESSION_TITLE_MAX (80) clean.slice(0, AGENT_SESSION_TITLE_MAX) cut(clean, AGENT_SESSION_TITLE_MAX).text
core/src/host/schedules.js:282 a schedule run's failure reason 200 reason.slice(0, 200) cut(reason, 200).text

cut is core/src/shared/text-boundary.js's helper, re-exported by core/src/host/text.js; each file imports it from ./text.js next to its other host helpers (hooks.js already imported it). None of the five maxima changes, and the neighbours the issue names — approvals.js:80, hooks.js:928/:939/:943, schedules.js:316 — are untouched.

generate re-made the mirrors: patches/ds-bot/0001-core-fixes.patch (the four core/src/host/ files that live in a topic) and the whole-file mirrors patches/ds-bot/files/src/host/hooks.js and patches/ds-bot/files/test/flybot.test.js; patches/ds-bot/topics.json gains test/agents.test.js under 0001-core-fixes, since that upstream file had no topic and generate refuses to write it blind.

Tests

Five cases, one per site, each driving the real entry point and each asserting the stored value is well formed, is the code-point prefix of the input, and — as a control — that a value under the bound is stored word for word:

site test file case
steward 80 core/test/steward.test.js the quoted first line of the kept tail is cut on a code point at 80
approvals 500 core/test/group-status.test.js an approval record's reason is cut on a code point at 500
hooks 240 core/test/flybot.test.js a mention receipt's reason is cut on a code point at 240
agents 80 core/test/agents.test.js a Session title is cut on a code point at the 80-unit bound
schedules 200 core/test/schedules.test.js a failed run's reason is cut on a code point at 200

Red first, each failing on its own. Run against the unfixed source of base 536f49f6 (those five files are byte-identical on 42b638c2 and 536f49f6), one file at a time:

[steward]       not ok 1 - P-1993: the quoted first line of the kept tail is cut on a code point at 80
[steward]         error: 'the quoted line is well formed: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\ud83d"'
[group-status]  not ok 1 - P-1993: an approval record's reason is cut on a code point at 500
[group-status]    error: 'the stored reason is well formed: "aaaa…aaa\ud83d"'   (499 a's + the lone high surrogate)
[agents]        not ok 1 - P-1993: a Session title is cut on a code point at the 80-unit bound
[agents]          error: 'the title is well formed: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\ud83d"'
[schedules]     not ok 1 - P-1993: a failed run's reason is cut on a code point at 200
[schedules]       error: 'the stored reason is well formed: "SERVER 503: aaa…aaa\ud83d"'   (187 a's after the prefix, then the lone high surrogate)
[flybot]        not ok 1 - P-1993: a mention receipt's reason is cut on a code point at 240
[flybot]          error: 'the receipt is well formed: "助手: not sent — aaa…aaa\ud83d"'   (239 a's, then the lone high surrogate)

Each case fails in its own file and on its own site; with all five fixed the same five cases are green, and the controls stay word for word.

Verification at the tip

Tip SHA: 29e3a3a (tree 3b3465a8dfaf9ca8a74076e86260d7dfe032724f, base cf810a7d1e6993f2609e97da604db07bf96f18ff)

Verified commit 9c6751e7a1ae80524fe7b7d3a02a091b2be45aad has that same tree; the pushed head above is the tree rebuilt on the base by the fork push helper, so the two commits differ only in parentage.

Ran, on a clean tree (git status --porcelain empty) at the verified commit:

TZ=UTC DSH_HARNESS=/home/xuke/.dsh/bot/workspace/a1727 \
  ARCH_DOCS_BASE=cf810a7d1e6993f2609e97da604db07bf96f18ff \
  npm_config_cache=/home/xuke/.dsh/bot/workspace/tmp-npm-cache npm run verify
node scripts/ds-bot-patches.mjs generate      # no change: `git status` after it is empty
node scripts/ds-bot-patches.mjs check         # 3 lines ok
node scripts/ds-bot-patches.mjs canonical     # "generate is a no-op"

verify exit 0: core 2394 tests / 2394 pass / 0 fail (base 2389 plus these five cases); plugins/social 541 (540 pass, 1 skip); plugins/tree 127 (126 pass, 1 skip); plugins/timeline 135 (134 pass, 1 skip); plugins/workflow 336; .github/scripts 563; interop 37; live-e2e 24 — 0 failures anywhere. arch-docs: OK (17 changed file(s)), satisfied by the two doc lines (docs/subsystems/core-host.md for the five host bounds, docs/subsystems/plugin-runtime.md for the mention receipt) rather than a Docs-not-needed: trailer. Lint: 0 errors (core 73 warnings, social 36, timeline 9, tree 2, workflow 16; interop 1 pre-existing warning at dev/interop/isolation/probe.mjs:130; live-e2e 0). The changelog fragment is changelog.d/1993-p2-c5-remaining.fixed.md.

Not run: the smoke test (core/dev/bench.sh smoke …) — it needs a free bench port and a DSH checkout on this box, and this change only swaps five string bounds whose paths the five new cases drive directly.

Checkpoint

{
  "pr": 2000,
  "claim": "Dev-2",
  "issue": 1993,
  "summary": "Five bounds in core/src/host cut text with a bare slice, so one landing between the halves of a surrogate pair stored a lone high surrogate that any encoder reads as U+FFFD. They were a checkpoint tail note's quoted first line (80), an approval record's reason (500), a mention receipt's reason line (240), a DSH Agent Session's title (80) and a schedule run's failure reason (200). Each now goes through the shared cut() with its maximum unchanged. Five cases, one per site, drive the real entry point and each was red on its own before the fix.",
  "files": [
    "core/src/host/steward.js",
    "core/src/host/approvals.js",
    "core/src/host/hooks.js",
    "core/src/host/agents.js",
    "core/src/host/schedules.js",
    "core/test/flybot.test.js",
    "docs/subsystems/core-host.md",
    "changelog.d/1993-p2-c5-remaining.fixed.md"
  ],
  "checks": {
    "ran": "TZ=UTC ARCH_DOCS_BASE=cf810a7d npm run verify; node scripts/ds-bot-patches.mjs check; canonical",
    "result": "green",
    "at": "29e3a3a5678e4b2be5cebf79adfaeadfd4d74051"
  },
  "next": "none — ready for review"
}

@HugoXK
HugoXK force-pushed the dev/2-p2-c5-remaining branch from f3ca07b to 29e3a3a Compare November 1, 2026 06:51
@Birfy

Birfy commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

Verdict: PASS at 29e3a3a
Written-by: Reviewer
Attestation: 29e3a3a role=Reviewer

Ran at this tip:

$ cp -a p2000tip p2000red && for f in agents approvals hooks schedules steward; do cp p2000base/core/src/host/$f.js p2000red/core/src/host/$f.js; done
$ cd p2000red/core && DSH_HARNESS=/opt/flybot node --import ./test/harness-resolve.mjs --test --test-concurrency=1 test/agents.test.js test/flybot.test.js test/group-status.test.js test/schedules.test.js test/steward.test.js   # the five changed sources swapped back to base blobs
# tests 134  # pass 129  # fail 5
$ cd p2000tip/core && DSH_HARNESS=/opt/flybot node --import ./test/harness-resolve.mjs --test --test-concurrency=1 test/agents.test.js test/flybot.test.js test/group-status.test.js test/schedules.test.js test/steward.test.js   # tip 29e3a3a5678e
# tests 134  # pass 134  # fail 0

Red first, and red exactly where it should be: on the base blobs the five new P-1993: cases fail one per site — Session title 80 (agents), mention receipt 240 (flybot), approval record 500 (group-status), failed run 200 (schedules), checkpoint tail note 80 (steward) — and the other 129 cases still pass. At the tip all 134 pass.

What I checked:

  • The five sites are the five the issue names, each swapping a bare slice for cut(...).text: core/src/host/steward.js:148, core/src/host/approvals.js:76, core/src/host/hooks.js:1041, core/src/host/agents.js:170, core/src/host/schedules.js:282. All five maxima are unchanged (80/500/240/80/200); hooks.js already imported cut from shared/text-boundary.js, the other four add it from ./text.js next to their other helpers.
  • cut is the shared boundary helper from core-host: decision and meeting archives cut text by code unit and store U+FFFD #1912/core-host: the remaining code-unit cuts (claim cue, read_own_chat query/around, page title, diagnostics lines, peer name) #1916/core: cut() stores a lone surrogate the input already carried #1963: it flattens with toWellFormed(), returns the longest code-point prefix within the same budget, and its .text is a prefix of the input, so a bound landing inside a surrogate pair drops the pair whole instead of storing a lone high surrogate.
  • Scope matches Fixes #1993: 17 files = 5 sources, 5 test files (one new case each), 2 doc lines (docs/subsystems/core-host.md "Since host: the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules) #1993", docs/subsystems/plugin-runtime.md receipt line), the changelog fragment, and the three regenerated patch mirrors (patches/ds-bot/0001-core-fixes.patch, files/src/host/hooks.js, files/test/flybot.test.js) plus topics.json — the mirror side of the change, not unrelated edits. compare cf810a7d0b...29e3a3a5 shows ahead 1 / behind 0, merge base = PR base.
  • CI at the tip: 13 of 14 check-runs green (tests-ok, merge-gate, package, judge, agent-audience ×2, plugins ×4, risk tier, reviewed tip, Node 24). The one failure is enqueue-guard, expected: it refuses this head only for "no Verifier report names the tip 29e3a3a" — this report is that missing piece.
  • Body metadata: Owner: Dev-2, Reviewer: Birfy (not the Owner), Written-by: Dev-2, checkpoint block present and within the 1,500/600 limits. No agent-pr label, and the guard says labels are not asked of a fork head, so none is needed.

Not run: npm run verify across all packages (the CI tests-ok job covers the suite at this tip); I ran the five affected files at tip and on the base blobs, which is what the five new cases drive.

@github-actions github-actions Bot added the pass label Nov 1, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 1, 2026
@Birfy Birfy added the agent-pr Pull request opened by an agent label Nov 1, 2026
Merged via the queue into ThinkFlowLab:main with commit b048e26 Nov 1, 2026
15 checks passed
@Birfy
Birfy deleted the dev/2-p2-c5-remaining branch November 1, 2026 07:03
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 1, 2026
@HugoXK

HugoXK commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

UTF-16 上限清单(归零审计,96eb3ef4)

Written-by: FlyBot Lead

这条是本 PR 修掉的那五处之外的同族剩余点,在 live main 96eb3ef4458ca0d0659a9e4584b5bae792c3b8af 上逐行读出来核的(35 张合入后的 tip,committer 15:36:49Z)。一行一个消费点,33 行 = A 7 / B 4 / C 14 / D 8;每行标「实测」(在 96eb3ef4 的代码上跑过)或「读码」(只读源码,没有独立入口)。

分组按改什么分,不按谁看:A 判据与切点一起改;B 只修切点、判据不动(「留多少」一字不变);C 单独议(wire 契约 / 模型输入 / 准入校验 / 预算);D 保持(已正确的写法,作对照)。

首行读数(实测)

core/src/host/text.js:61 的 clip 用 text.length > max(UTF-16 单元)判超限、用 codePointEnd(码点)切:

输入 现在 修后(按 fits 推)
clip('😀'.repeat(300), 400, 'r') 200 个 emoji + … (cut; r)(410 units / 210 码点) 原样返回(300 字符 ≤ 400)
clip('😀'.repeat(201), 400, 'r') 200 个 emoji(砍 1 个) 原样返回
clip('😀'.repeat(201), 200, 'r') 100 个 emoji 恰好 200 码点
excerpt('😀'.repeat(300), '', 400) 200 个 emoji + … 原样返回

也就是一段 300 个字符的文本,被一个 400 个字符的上限砍掉三分之一。这条经 read_own_chat 的 list / around / handoff 行直接给到用户和 Bot 自己,parts.js:344/:355/:465 与 chat-index.js:213/:310 都是它的消费点。

A — 判据与切点一起改(7)

文件:行 现在的判定 建议 面
core/src/host/text.js:61(clip) 实测 单元判(text.length > max)、码点切(codePointEnd) 判据改 fits,切点不动 显示 —— 清单首行
core/src/host/text.js:47(excerpt) 实测 同样单元判、码点切;max 默认 OWN_CHAT_LINE_MAX = 400(host/constants.js:72) 判据改 fits 显示(搜索结果行)
core/src/host/parts.js:344 读码 无自有判定,clip(line.text, RECENT_LINE_MAX, …) 随 text.js:61 显示(recent 行)
core/src/host/parts.js:355 读码 clip(line.text, LIST_LINE_MAX, …) 同上 显示(listed 行)
core/src/host/parts.js:465 读码 clip(line.text, RECENT_LINE_MAX, …) 同上 显示(handoff keptMessages)
core/src/host/chat-index.js:213 读码 rowLine 里的 excerpt(text, wanted, max) 随 text.js:47 显示(搜索行)
core/src/host/chat-index.js:310 读码 lines.push(... excerpt(line.text, wanted)) 同上 显示(搜索命中)

B — 只修切点、判据不动(4)

文件:行 现在的判定 建议 面
core/src/host/steward.js:49 实测 单元判 text.length <= STEWARD_LINE_MAX;head = ceil(1500·2/3) = 1000,头尾都用 slice,head 落进代理对就留孤立代理 两个切点换 cut(...).text,判据与 head 算法原样 模型输入(checkpoint 视图)
core/src/host/steward.js:51 实测 省略数 text.length - STEWARD_LINE_MAX 是 units(同一段 2602-unit 输入显示 1102 characters left out) 同一处计数换码点 模型输入(提示文案)
core/src/host/plugin-model.js:64 读码 + 算术实测 String(input.system).slice(0, caps.maxSystemChars),单元切、无边界回退 改 cut(...).text 模型输入(系统提示)
core/src/client/diagnostics.js:21、:40、:92、:93 读码 + 算术实测 四处裸 slice:error.message 300、target/from 60 与 stack 300、what.trim() 2000、title 80 四处改 cut(...) 切 显示(诊断条目与 bug 报告预填文本)

B 组复现(每处都把原表达式原样搬进探针):

  • steward:49 —— 'a'*999 + 😀 + 'b'*1601(2602 units),head = 1000 落在 😀 前半:头段以孤立高代理收尾,UTF-8 往返后 1 个 U+FFFD。
  • plugin-model:64 —— maxSystemChars = 100,'a'*99 + 😀 + 'b'*50:slice(0,100) 以孤立高代理结尾,UTF-8 往返 1 个 U+FFFD。
  • client/diagnostics:92 —— WHAT_MAX = 2000,'a'*1999 + 😀 + ' tail':slice(0,2000) 以孤立高代理结尾,拼进 URLSearchParams 后 URL 里出现 %EF%BF%BD(GitHub 预填正文里就是「�」)。

C — 单独议(14)

C1 wire 契约(6) —— 改单位会改对端看到的字节边界

文件:行 现状
core/src/shared/room-share.js:58 wire().length > ROOM_SHARE_TEXT_MAX(裁 history)
core/src/shared/room-share.js:88 wire().length(notice)
core/src/shared/room-share.js:94 wire().length(notice 放不下就删)
core/src/shared/room-share.js:104 body.text.length + wire().length(text 循环)
core/src/shared/room-share.js:115 out.length(最终断言)
core/src/host/room-membership.js:1041 text.length < 1 || text.length > ROOM_SHARE_TEXT_MAX(对端报 1–4000 chars)

C2 模型输入(2)

文件:行 现状
core/src/host/memory.js:1270 本地重写的 clip:text.length > max ? cut(text,max).text + ' …(cut)' : text —— 与 text.js:61 同病同形,只差措辞
core/src/host/memory.js:1272-1278 ends:尾段走 slice(codePointStart(...))、头段走 cut,切点已码点安全,只有判据在单元上(只多截)

C3 准入与校验(3) —— 改单位会改「收不收」

文件:行 现状
core/src/host/plugin-model.js:63 prompt.length > caps.maxPromptChars → prompt-too-long(只判不收)
core/src/host/secrets.js:255 value.length < MIN_VALUE,文案写 "characters";改单位会让同一把旧 key 从可存变不可存
core/src/host/secrets.js:256 value.length > MAX_VALUE,同上(只提措辞)

C4 预算大小(3) —— 只影响多留/少留几行

文件:行 现状
core/src/host/room-context.js:139-140 used + gap + text.length > budget 判单元;:143 截断走 cut(text, room-1),切点已安全
core/src/host/memory.js:266 used + entry.text.length > budget(记忆索引预算)
core/src/host/memory.js:1355 used + text.length > REVIEW_TEXT_MAX(回顾批次预算)

D — 保持(8)

文件:行 现状 为什么保持
core/src/host/busy-ack.js:72 [...clean].length > ACK_MAX_CHARS 码点判
core/src/host/hooks.js:734 codePoints(text) > MAX_SEND_TEXT 码点判(#1895)
core/src/shared/remote-label.js:8 cut(text, 60).text 码点切
core/src/host/diagnostics.js:52-53 cut(out, TEXT_MAX) + kept.cut 码点切、有 cut 标志
core/src/host/room-context.js:284 lines.join('\n').length > ROOM_PRESENCE_BUDGET 整行增删,不在字符区间切
core/src/client/cells.js:181(+:184) line.length <= PEEK_MAX 判单元,但按 grapheme 累加 不会劈代理对,只多截
core/src/host/project-assistant.js:134 title.length > 80 判单元,切用 cut(title, 77) 只多截(切点已由 #2005 修好)
core/src/client/details-drawer.js:96 text.length > 240 决定是否折叠 不切字符,只影响「展开」是否出现

两处容易混的

  • 两个 diagnostics.js:core/src/host/diagnostics.js:52 是 cut(out, TEXT_MAX) + :53 kept.cut,已经码点安全 → D;core/src/client/diagnostics.js 的 :21/:40/:92/:93 是裸 slice → B。按文件名搜会撞在一起,按路径看。
  • room-context.js 两处不是一回事::139-140 判单元、:143 已用 cut 截(C4);:284 是整行预算、不切字符(D)。

下一步

  • core/src/shared/text-boundary.js 已有 codePointStart / codePointEnd / cut(cut 先 toWellFormed(),返回 {text, cut});缺的是 fits(text, max)(= 码点数 ≤ max)。A 组用它判、仍用 codePointEnd / cut 切;B 组只换切点。做成一条表驱动检查(「上限 → 消费点」),新加一处按单元的判定就以「文件:行」报红——那张单开好后在本评论补链接。
  • C 组本轮不动:wire 改单位会改对端字节边界,准入/校验改单位会改「收不收」,预算改单位会改可容纳行数——三类都要单独判。
  • 本 PR(host: the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules) #1993)修的五处都在这族的切点半边;剩下的是判据半边,D 组是已经正确的写法。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

host: the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules)

2 participants