diff --git a/plugins/avatasia/mmc-trajectory/.gitignore b/plugins/avatasia/mmc-trajectory/.gitignore index 1e95f573..1b285758 100644 --- a/plugins/avatasia/mmc-trajectory/.gitignore +++ b/plugins/avatasia/mmc-trajectory/.gitignore @@ -11,4 +11,10 @@ __pycache__/ # mid-mutation is caught on the next one instead of being inferred from whatever the source looks # like afterwards. The snapshot is machine-specific bookkeeping and this directory is the package # that ships to readers, so it must never be tracked. -tests/.shipped-snapshot.json \ No newline at end of file +tests/.shipped-snapshot.json + +# Derived artifacts. The runtime writes the search segments under its own data dir, which is +# outside this package; the runtime refuses a data dir that lands in here (see `start()` in +# miniapp/node/server.mjs). This entry is the second half of that rule: if a caller ever got +# through, the directory must not be tracked and must not be part of what ships. +trajectory-artifacts/ \ No newline at end of file diff --git a/plugins/avatasia/mmc-trajectory/.minimax-plugin/plugin.json b/plugins/avatasia/mmc-trajectory/.minimax-plugin/plugin.json index 32477bb8..ac0370bf 100644 --- a/plugins/avatasia/mmc-trajectory/.minimax-plugin/plugin.json +++ b/plugins/avatasia/mmc-trajectory/.minimax-plugin/plugin.json @@ -2,7 +2,7 @@ "schemaVersion": 1, "name": "mmc-trajectory", "displayName": "会话轨迹", - "version": "1.2.0", + "version": "1.3.0", "description": "按轮次浏览当前会话的模型轨迹：消息流、思考过程、工具调用与结果、token 与耗时统计。", "author": "avatasia", "icon": "icon.png", diff --git a/plugins/avatasia/mmc-trajectory/README.md b/plugins/avatasia/mmc-trajectory/README.md index 618e6cee..3c64fbce 100644 --- a/plugins/avatasia/mmc-trajectory/README.md +++ b/plugins/avatasia/mmc-trajectory/README.md @@ -4,7 +4,7 @@ English | [简体中文](README.zh-CN.md) Browse the model trajectory of a MiniMax Code conversation: messages, reasoning, tool calls and their results, Token usage, and per-turn timing. -Author: [avatasia](https://github.com/avatasia) · Version: `1.2.0` +Author: [avatasia](https://github.com/avatasia) · Version: `1.3.0` ### Before you change this page @@ -131,7 +131,8 @@ The runtime database is opened in read-only mode. If it is missing, unreadable, - A single `messages.jsonl` is read up to 64 MB. Larger sessions are truncated and the page says so. - The ledger does not use virtual scrolling; every rendered row is a real element. The page therefore opens on the newest 200 records, and the window bar button switches to the whole session. The button names what pressing it will do, not the current state: 「显示全部」 at the default and 「显示最新 200 条」 once everything is showing, while the count beside it reports what is on screen. That full view is opt-in for a reason: on a 9,900-record session it is roughly 57,000 elements rebuilt every 2.5s poll, which was measured to freeze the page rather than be read. 「向上加载更早的 N 条」 widens the window in steps and a poll does not undo it. A sticky window bar under the overview strip offers that choice behind four explicit controls: switch the strip between equal width and a real time axis, load 200 earlier records, jump back to the latest 200, and show everything. "Show everything" builds every record in the session into the DOM at once and gets noticeably slower on large sessions; incoming records do not knock it back down to 200. - `messageCount` and `turnCount` in the session picker are estimates derived from a bounded prefix of the file; exact values come from the selected session. -- A session the runtime spawned from another one is marked 「子会话」 in the picker. The parent is deliberately not named: a branch is always newer than the session that spawned it, so its parent has usually aged out of the most-recent-N window this list is built from, and a lookup that fails quietly is how a picker ends up asserting something it cannot see. +- The session picker is **two selects in a cascade**. The upper one, 「会话」, lists conversations only — the sessions whose parent, followed up the chain, is a conversation. A session with no parent is one of them, and so is one whose parent names no session, whose parent chain ends in a cycle, or whose parent is itself a run: the layer follows the chain rather than the one column, and a row that ends up in the conversation layer carries no parent. The lower one, 「子代理」, lists the runs of the conversation that was picked: it is **empty by default**, it has values once a conversation is picked, and it is empty and disabled while the page follows the newest conversation. Measured on this machine, 13 of the 20 rows were sub-agent runs, so a reader could not tell which row was their own conversation; those 13 runs were also where that conversation actually happened. Nothing is hidden, only moved — the runs are not rows of the conversation list any more, they are rows of the run list under it, and the conversation list is now the twenty most recent conversations rather than the twenty most recent sessions of any kind. Every run is listed, most recently active first, and pressing one switches the page to it. The layer is decided by the chain of parents, which is independent of the 「定时 / 任务 / 轻量」 markers. The parent's name is still not written out: the parent id comes down with the row, and the page groups by that one id rather than doing a lookup that could fail quietly. Picking a run shows **that run's own trajectory** — a run is a session with its own transcript, and the parent's transcript does not contain its turns — and the conversation select stays on its parent. +- The picker also says what a session **is**, not only what it is called: a scheduled run carries 「定时」, a background task 「任务」, and a lightweight conversation 「轻量」 — the markers come from the runtime's own record of the session, not from anything this app checks — so none of them can be mistaken for a conversation you wrote. They stay in the list and stay selectable — a marker is a marker, not a filter. The conversation the runtime database reports as current comes first, then the sessions of its own workspace, then the rest; the workspace path and the raw purpose string are compared on the server and never reach the page. - Light theme token coverage is verified — every `--mcode-*` token the page consumes is defined for both themes — but its rendered appearance was not visually checked; the page was exercised under a dark system preference. ## Tested environment diff --git a/plugins/avatasia/mmc-trajectory/README.zh-CN.md b/plugins/avatasia/mmc-trajectory/README.zh-CN.md index eab669ce..29eb0f4c 100644 --- a/plugins/avatasia/mmc-trajectory/README.zh-CN.md +++ b/plugins/avatasia/mmc-trajectory/README.zh-CN.md @@ -4,7 +4,7 @@ 按轮次浏览一段 MiniMax Code 对话的模型轨迹：消息流、思考过程、工具调用与结果、token 消耗与每轮耗时。 -作者：[avatasia](https://github.com/avatasia) · 版本：`1.2.0` +作者：[avatasia](https://github.com/avatasia) · 版本：`1.3.0` ### 动手改这个页面前 @@ -117,7 +117,8 @@ MiniMax Code 不会告诉 Mini App 是哪段对话打开了它的页面：传给 - 单个 `messages.jsonl` 最多读取 64 MB。更大的会话会被截断，页面会明确说明。 - 台里没有用虚拟滚动，每一行都是真实的 DOM 元素。页面默认只展示最新 200 条，窗口条上的按钮可以切到全量；全量是可选的，因为它很贵 —— 在这个会话上约 9,900 条、约 5.7 万个元素，每 2.5 秒轮询重建一次，实测会直接卡死页面。总览条正下方的吸顶窗口栏提供三个显式控件：「总览轴」「向上加载更早的 200 条」，以及在最新 200 条与全量之间切换的那一个按钮（它写的是按下去会变成什么，默认为「显示全部」，全量时为「显示最新 200 条」；当前窗口由右边的计数条报告，两者回答的是不同问题）。「总览轴」只改总览条的横轴——等宽（每条记录一样宽，完全忽略时间）或真实时间轴（宽度等于该记录的真实耗时）；它不筛选、不排序，台账里的行和页面上的任何计数都不变。「向上加载更早的 200 条」每点一次就多加一页，轮询不会把已经加宽的窗口重置回去；点切换按钮也不会自动滚动页面，读者正在看的那一行会留在原地。 - 会话选择器里的 `messageCount` / `turnCount` 是从文件的有界前缀估算的；精确值以选中会话后展示的为准。 -- 会话选择器里，被运行时从另一个会话派生出来的会话标为「子会话」，但不写父会话的名字：分支一定比派生它的会话新，而列表按最近活动取前 30 个，父会话通常已经掉出这个窗口。静默失败的查找，最终会让列表声称一个它看不见的东西。 +- 会话选择器是**级联的两个下拉框**。上面那个「会话」只列主会话 —— 沿父链往上走、终点是一段主会话的那些。父会话为空的是主会话；父会话指向一个不存在的会话、父链成环、或者父本身是子代理的，也是主会话 —— 层级看的是链，不是那一列，而落到会话层的行不带父会话。下面那个「子代理」只列所选那段对话的子代理，**默认是空**，选中主会话之后才有值，跟随最新时是空的且不可选。实测本机 20 行里有 13 行是子代理，读者根本分不出哪一行是自己的对话，而那 13 行恰恰就是这段对话真正发生的地方。它们没有被藏起来，只是从「平铺在会话里」变成「会话列表下面的子代理列表」：会话列表现在列的是最近 20 段主会话，而不是最近 20 个任意种类的会话；子代理有多少列多少，按各自最后一次活跃的时间倒序，最活跃的排在最上面，点其中一条就切到那一条。层级由父链决定，与「定时 / 任务 / 轻量」那套标记互相独立。父会话的名字仍然不写：父 id 由服务端随这一行带下来，页面按同一个 id 归组，不做一次可能悄悄失败的查找。选了子代理就显示**那一次运行自己的**轨迹 —— 每个子代理是一段独立的会话，有自己的 transcript，父会话的轨迹里并不包含它的轮次；此时上面那个下拉框停在它的父会话上。 +- 会话选择器不只说会话**叫什么**，还说它**是什么**：定时任务标「定时」，后台任务标「任务」，轻量会话标「轻量」（标记来自运行时对会话自己的记录，不是这个应用检查出来的），于是它们不会被误认成你自己开的对话。它们仍然在列表里、仍然可以选 —— 标记是标记，不是过滤。运行时库报告为当前的那段对话排在最前，然后是同一工作区的会话，再然后是其它；工作区路径和 purpose 原文只在服务端比较，不会发到页面。 - 浅色主题的 token 覆盖已验证（页面用到的每个 `--mcode-*` 变量在两套主题下都有定义），但渲染外观未做视觉检查；功能验收是在深色系统偏好下进行的。 ## 测试环境 diff --git a/plugins/avatasia/mmc-trajectory/miniapp/client/index.html b/plugins/avatasia/mmc-trajectory/miniapp/client/index.html index b54c571b..299d0432 100644 --- a/plugins/avatasia/mmc-trajectory/miniapp/client/index.html +++ b/plugins/avatasia/mmc-trajectory/miniapp/client/index.html @@ -1409,6 +1409,48 @@ content-visibility: auto; } + /* A group that is holding its height but not its rows. + + The rows are gone, so the box is empty — and an empty box is a box of no height, which is + how the document under the reader loses the height of everything they have scrolled past. + The height is written onto the group from JavaScript, from the height it had while it was + holding its rows — brought up to date with what the group holds now, since the turn keeps + its key while its content changes — and it is the only thing that holds the box while the + browser is RENDERING the group rather than skipping it: `contain-intrinsic-size` sizes an + element the browser has skipped, and a group near the viewport is rendered. + + Measured, on a real session of 47,937 records: releasing the rows of the groups around + the viewport took the scroll height down by 79,288 px, and releasing the viewport's own + group took another 257,378 px. Pinned by tests/verify-ledger-lifecycle.mjs. */ + .turn-group.is-released .row-list { + min-height: 2px; + } + + /* The group is on its way back. The bar stands where the rows are coming, so the reader + sees a line rather than a hole; the group's own height is held by its min-height, so the + bar costs the document nothing. */ + .turn-group.is-released .row-list::after { + content: ''; + display: block; + height: 2px; + margin: 10px 0; + border-radius: 1px; + background: var(--mcode-accent, currentColor); + opacity: .3; + animation: ledger-release 1.2s ease-in-out infinite; + } + + @keyframes ledger-release { + 0%, 100% { opacity: .1; transform: scaleX(.3); } + 50% { opacity: .55; transform: scaleX(1); } + } + + @media (prefers-reduced-motion: reduce) { + .turn-group.is-released .row-list::after { + animation: none; + } + } + .turn-head { position: sticky; /* Below every band above it, measured. It used to pin at 0 with z-index 2 and simply got @@ -1496,15 +1538,15 @@ digits — the phone behaviour. */ :root { --traj-index-col: 72px; - /* The docked height, as two fifths of the window. It was 50vh, then a quarter, and the - unit was never the problem — the unit being frozen was. The reader cannot make it - smaller, and a cap the reader cannot move is a number nobody chose. applyInspectorDock() - therefore writes this in px from window.innerHeight and a drag overwrites it, so the - declaration below is only what shows before the first measurement. It stays a vh for the - same reason as everywhere else here: 100vh is not the reader's height inside a host - iframe, and every offset on this page is measured for that reason. The 40 must track - INSPECTOR_DOCK_SHARE — the fallback and the measurement are the same promise written - twice, so a guard holds them together. */ + /* The docked height, as the share of the window that INSPECTOR_DOCK_SHARE declares. It + was 50vh, then a quarter, and the unit was never the problem — the unit being frozen + was. The reader cannot make it smaller, and a cap the reader cannot move is a number + nobody chose. applyInspectorDock() therefore writes this in px from window.innerHeight + and a drag overwrites it, so the declaration below is only what shows before the first + measurement. It stays a vh for the same reason as everywhere else here: 100vh is not + the reader's height inside a host iframe, and every offset on this page is measured + for that reason. The declared vh must track INSPECTOR_DOCK_SHARE — the fallback and + the measurement are the same promise written twice, so a guard holds them together. */ --inspector-dock-h: 40vh; } @@ -2198,13 +2240,13 @@ The cap stays on this rule, because it is a property of every state and not of the pinned ones. It is `--inspector-full-h`, the same number `is-inspector-full` uses, - because that is already `innerHeight - pinned strips - the ledger's reserved quarter`: - the panel can be neither taller than the window nor able to squeeze the rows. What - changed is the reason. It used to be 「a sticky-bottom element has nowhere legal to - sit」, which was true while sticky was here and is not true now; in flow this is an - ordinary static block. It is capped because the panel is the last thing in the - document, and a panel taller than the window is one the reader has to scroll past to - reach the end of the page. */ + because that is already `innerHeight - pinned strips - the ledger's reserved share` + (INSPECTOR_LEDGER_SHARE): the panel can be neither taller than the window nor able to + squeeze the rows. What changed is the reason. It used to be 「a sticky-bottom element + has nowhere legal to sit」, which was true while sticky was here and is not true now; + in flow this is an ordinary static block. It is capped because the panel is the last + thing in the document, and a panel taller than the window is one the reader has to + scroll past to reach the end of the page. */ #inspector-panel { max-height: var(--inspector-full-h, 70vh); overflow-y: auto; @@ -2222,16 +2264,16 @@ z-index: 5; } - /* Two fifths by default, and the reader can drag it. The content scrolls inside the - panel instead of the panel growing: a growing panel pushes its own handle out of the - viewport, which is the one thing a docked panel cannot do. The number itself is written - by applyInspectorDock() — a fifth of the measured window twice over, or whatever the reader - dragged to — so this rule only says what the cap is *called*. + /* Docked at INSPECTOR_DOCK_SHARE by default, and the reader can drag it. The content + scrolls inside the panel instead of the panel growing: a growing panel pushes its own + handle out of the viewport, which is the one thing a docked panel cannot do. The number + itself is written by applyInspectorDock() — INSPECTOR_DOCK_SHARE of the measured window, + or whatever the reader dragged to — so this rule only says what the cap is *called*. `max-height`, never `height`, and that is the answer to 「窗口的高度和检视的高度取 最大还是最小」: a cap renders min(content, cap), so a record whose body is shorter - than the cap is shown at its own height instead of being padded out to a fifth of - the window. `height` would make the cap mandatory and put a blank band inside the + than the cap is shown at its own height instead of being padded out to the docked + height. `height` would make the cap mandatory and put a blank band inside the panel. It cannot show today — the shortest record measured 752px of content against a 400px cap, because the summary tab always renders its metadata table — which is exactly why it is a rule worth writing down rather than a coincidence worth relying on. */ @@ -2244,8 +2286,8 @@ /* Folded down to its own header. This is the third height the reader can pick, and it is the one that was missing: `flow` and `docked`/`full` are three class values, but only two - of them are heights — flow is a *position* ("not pinned"), so the panel could go from two - fifths to fully expanded and had nowhere smaller to go. + of them are heights — flow is a *position* ("not pinned"), so the panel could go from the + docked height to fully expanded and had nowhere smaller to go. Folded by taking the content out of flow rather than by capping the panel. The obvious version is `max-height: `, and that is the version this section @@ -2456,6 +2498,14 @@

会话轨迹

+ +
+ + +
@@ -2813,7 +2863,7 @@

检视

// A turn that took a prompt and never produced a reply — the shape an interruption // leaves behind. The turn still running is excluded: it is open because it is // going, which is not the same thing as broken, and a filter that re-flags the live - // turn on every 2.5s poll is a filter nobody can leave on. + // turn on every poll is a filter nobody can leave on. turn: function (turn, ctx) { return !!turn && turn.closed === false && turn !== ctx.liveTurn; } @@ -2829,8 +2879,7 @@

检视

// It keeps its kind on purpose. The record genuinely is a reply, the 答复 chip counts // it and the reply filter keeps it — it is an anomaly *as well as* a reply, not // instead of one. 无答复 cannot see it either way: an empty reply closes its turn, so - // `closed` is true and that rule has nothing to fire on. Measured on this session: - // two, one of them the turn that ended in front of the reader with nothing shown. + // `closed` is true and that rule has nothing to fire on. // `hasText` is the server's own verdict, measured from the text it projected, and it is // NOT the same question as whether `summary` is empty: a body-less reply is given its // title as a summary, so reading one from the other would find nothing to flag. @@ -3237,6 +3286,50 @@

检视

return flat.slice(0, Math.max(0, limit - 1)) + '…'; } + /* The badge that says what KIND of session a row is. + * + * The service sends a token, never the raw `purpose` it read from the runtime database: + * that value is free text and can carry anything a person typed into it, so what travels + * is a closed set of three words and nothing else. An unknown token is not a badge — a + * token this page does not know about must not become a label it invented. + * + * It is a marker, not a filter: a scheduled run is still selectable, the reader is just + * told what it is instead of being handed something that looks like their own + * conversation. */ + function sessionTagBadge(tag) { + if (tag === 'lightweight') return ' · 轻量'; + if (tag === 'scheduled') return ' · 定时'; + if (tag === 'task') return ' · 任务'; + return ''; + } + + /** + * The three parts of a session's label, in the order they are printed. + * + * One function, because two places print a session's label — the picker's option and the + * sub-agent row under it — and two copies of a format is two places for the format to + * drift. It returns the parts rather than the whole string because the two callers put + * them in different shapes: the picker appends 「子会话」 after them, and the sub-agent row + * would only be repeating itself. + * + * DOM-free on purpose: it lives in this block so the offline probe can run it without a + * page, and so a change to the format is one edit rather than two. + * + * @param {any} session + * @param {number} now + * @returns {{ label: string, tail: string, tag: string }} + */ + function sessionLabelParts(session, now) { + var label = truncateText(session && session.label ? session.label : '', 42); + var when = formatAgo(session ? session.lastActiveAt : null, now); + var count = session ? session.messageCount : null; + var tail = isCount(count) ? '（' + when + ' · ' + formatInt(count) + ' 条）' : '（' + when + '）'; + // What kind of session this is, when the service said. It is a marker and not a filter: + // the row is still selectable, and a row with no kind of its own gets no marker. + var tag = session ? sessionTagBadge(session.sessionTag) : ''; + return { label: label, tail: tail, tag: tag }; + } + function safeText(value) { if (value === null || value === undefined) return ''; if (typeof value === 'string') return value; @@ -3378,7 +3471,7 @@

检视

} /** - * Cheap change detector: rebuilding 200+ rows every 2.5s is wasteful + * Cheap change detector: rebuilding a window of rows on every poll is wasteful * when nothing changed. Only structural counters participate. */ function signatureOf(payload) { @@ -3401,7 +3494,7 @@

检视

].join('|'); } - /** markdown handed to the Host chat input; raw JSON capped at 8000 chars. */ + /** markdown handed to the Host chat input; raw JSON capped at HANDOFF_RAW_LIMIT chars. */ var HANDOFF_RAW_LIMIT = 8000; function truncateJson(value, limit) { @@ -3486,7 +3579,7 @@

检视

function inTimeRange(record) { if (!recordInRange(record, state.rangeFrom, state.rangeTo)) return false; // The snapshot half of the range. Without it an open-ended range grows on its own and - // the table appends itself every 2.5s. + // the table appends itself on every poll. if (isCount(state.rangeCutoff) && isCount(record.index) && record.index > state.rangeCutoff) { return false; } @@ -3610,8 +3703,8 @@

检视

// Two cadences, one per fetch, because the two are not the same work. See startPolling(). // - // 2500 for the list, which is a cached walk and a cached title read and now usually a 304 — - // and which the picker renders 「刚刚」 against. 8000 for the trajectory, which reparses a + // SESSION_POLL_INTERVAL for the list, which is a cached walk and a cached title read and now usually a 304 — + // and which the picker renders 「刚刚」 against. TRAJECTORY_POLL_INTERVAL for the trajectory, which reparses a // lineage and reopens the runtime database on every request. The list wakes the trajectory // early when it has evidence, so the slow tick is a floor on latency, not a replacement for // noticing a change. @@ -3631,6 +3724,10 @@

检视

var state = { sessions: [], + // The sub-agents of the main sessions in `sessions`, flat: every row names the main it + // hangs off. One list and one join key, rather than a second hierarchy — the page groups + // them by parent, and the service decides who the parent is. + children: [], sessionsLoaded: false, followLatest: true, // Seeded from the host id so the picker and the content agree. `sessionParam()` already @@ -3661,7 +3758,7 @@

检视

// which the dock has moved on and the memory has not. inspectorDockBeforeFold: 'docked', // The docked height in px, once the reader has dragged it. null means "they have not", and - // the quarter-window default applies. A separate field rather than an extra mode on + // the dock's default height applies. A separate field rather than an extra mode on // inspectorDock because it survives the state leaving the dock: coming back to the panel // later should not silently forget the height the reader just chose. inspectorDockH: null, @@ -3716,6 +3813,24 @@

检视

// Why the last search could not be answered, or null. 「没搜到」 and 「没搜成」 are // different states and the count line has to be able to tell them apart. searchError: null, + // How many times this needle has been re-asked against a freshly fetched body because the + // answer turned out to be about a different version of the records. Bounded, because a + // session that is being written to keeps moving — the counter is what stops the page from + // chasing it forever, and it is reset by the next answer that does apply. + searchRefresh: 0, + // The needle `searchRefresh` belongs to. A new needle is a new question and starts over. + searchRefreshFor: '', + // WHICH SET the search answer on screen is about: the session and the slice of the body it + // was computed against, and a fingerprint of that body's records. + // + // A body can change without changing its shape — a record's text rewritten under the page + // keeps every counter the same — so the shape of the body cannot say whether the answer + // still covers the records on screen. This does: it is the same set the request names, so + // the standing answer and the request cannot disagree about which records it is about. + // + // Written in exactly one place: `runSearch`, when the question is put. Read in exactly one + // place: `applyPayload`, which is the only writer of the body on screen. + searchSet: '', // Time range, in milliseconds since the session started — the same zero the ledger's // `+893 分 21 秒` column counts from. `null` on either side means that end is open, // and both null is the default: no range, whole session. @@ -3742,13 +3857,13 @@

检视

rangeToText: '', limit: PAGE_STEP, // 「全部」 is a mode, not just a big `limit`. A poll that arrives mid-session used to - // re-pin the window to the newest page, which would have thrown 「全部」 back to 200 + // re-pin the window to the newest page, which would have thrown 「全部」 back to the default window // the instant a record landed. This flag is what says "the reader asked for all of // it", so a growing session extends the view instead of cancelling it. // // It opens OFF, and that is not a shortcut: 全部 renders every record of the session as // real DOM — around 9,900 rows and 57,000 elements on this one — with no virtual - // scrolling, rebuilt on every 2.5s poll. On load that froze the page outright (a single + // scrolling, rebuilt on every poll. On load that froze the page outright (a single // inspect took 13.7s, a click 5.4s), which was measured, not estimated. So 全部 is // something a reader asks for and can live with, not something the page does to them. allRecords: false, @@ -3804,7 +3919,7 @@

检视

* The one writer for this layer, and it reports whether the bar's state actually moved. * * That return value is load-bearing rather than a convenience: `render()` rebuilds the - * whole ledger, and a quiet session revalidates into a 304 every 8 seconds, so a fix that + * whole ledger, and a quiet session revalidates into a 304 on every poll, so a fix that * re-rendered on every 304 would trade a sticky banner for a permanent rebuild. * * @param {'trajectory'|'sessions'} source @@ -3861,6 +3976,9 @@

检视

var dom = {}; var inflight = null; var sessionsInflight = null; + // The sequence that decides which of several list requests is allowed to answer. The + // controller above says which request to stop; this says which answer may write. + var sessionsSeq = 0; var sessionPollTimer = null; var trajectoryPollTimer = null; var searchTimer = null; @@ -3885,6 +4003,7 @@

检视

dom.kindControls = document.getElementById('kind-controls'); dom.filterSticky = document.getElementById('filter-sticky'); dom.sessionSelect = document.getElementById('session-select'); + dom.childSelect = document.getElementById('child-select'); dom.generationField = document.getElementById('generation-field'); dom.generationSelect = document.getElementById('generation-select'); dom.generationHint = document.getElementById('generation-hint'); @@ -3972,6 +4091,206 @@

检视

return state.followLatest || !state.sessionId ? 'latest' : encodeURIComponent(state.sessionId); } + /** + * The session the body on screen is about, for a read that addresses a position inside it. + * + * A record index is a position within one body, so a request carrying that index has to ask + * about the session that body was built for. `sessionParam()` answers a different question + * — which session is current — and while the page follows the newest conversation that is + * `latest`, which is not necessarily the body on screen: the runtime can have made another + * session current while the page still shows this one. + * + * The client's own cache key already says which body a value belongs to — `rawCacheKey` + * names the displayed session — so asking the server for `latest` stored one session's + * record under another session's key, and the page showed the other session's words. + * + * @returns {string} + */ + function displayedSessionParam() { + var payload = state.payload; + var session = payload && payload.session ? payload.session : null; + var sid = session && typeof session.id === 'string' ? session.id : ''; + if (sid) return encodeURIComponent(sid); + return sessionParam(); + } + + /** + * The slice of history the body on screen is, for a request that addresses a position + * inside it. + * + * A record index is a position within one body, and a body is one slice of one session's + * history. The reader's SELECTION and the body on screen are two different things: the + * reader can pick a generation, and the load of that generation can fail and leave the + * previous body on screen. A request built from the selection would then ask about a slice + * the page is not showing, and the answer's indexes would filter rows that are not there. + * + * So this reads the slice the payload says, which is the slice it was built for. The + * service sends it as the payload's own generation, and it means the same thing there as + * it does in a search answer: `null` is the whole lineage. + * + * @returns {number | null} + */ + function displayedGeneration() { + var payload = state.payload; + var session = payload && payload.session ? payload.session : null; + return session && typeof session.generation === 'number' ? session.generation : null; + } + + /** + * The records of the body on screen, by their index. + * + * A record index is a position inside ONE body, so anything that names an index has to be + * able to say which record of that body it means. This is the same set the ledger draws — + * the flattened body, not a second copy of it. + * + * @returns {Map} + */ + function displayedRecordsByIndex() { + var map = new Map(); + var flat = Array.isArray(state.flat) ? state.flat : []; + for (var i = 0; i < flat.length; i++) { + var entry = flat[i]; + var record = entry && entry.record ? entry.record : entry; + if (record && typeof record.index === 'number') map.set(record.index, record); + } + return map; + } + + /** + * The identity of a SET of records: a fold over their version tokens, in the order they are + * shown. + * + * A version token says 「this one record is this one version」. A search answer is about a SET + * of them — it says 「these match and the rest do not」 — and the part that matters to the + * reader is the part it does NOT name. So the set needs an identity of its own, and it has to + * be one the service can compute over ITS records and compare. + * + * Two properties, and both of them are the point: + * + * - **Stable under APPEND.** A session that grows does not change the records it already + * has, so the fold over the first N records is the same before and after an append. A + * body-level identity (an ETag, a digest of the whole body) changes every time the session + * is written to, and then every answer stops applying every few seconds — the common case + * would be the broken one. + * - **Different when a record is rewritten.** The token is a digest of that record's own + * bytes, so a record whose text was replaced under the page has a different token, and the + * fold over the same positions has a different answer. + * + * The fold is ordered, so a set that was reordered is a different set. Two lanes of 32 bits + * each. This is the same function as `recordSetFingerprint` in the service, and + * `tests/client-check.mjs` pins that the two agree — two implementations of one question is + * two answers, and the difference between them is nobody's to explain. + * + * @param {Array} versions + * @returns {string} 16 hex characters + */ + function recordSetFingerprint(versions) { + var h1 = 0x811c9dc5 >>> 0; + var h2 = 0x9e3779b9 >>> 0; + var list = Array.isArray(versions) ? versions : []; + for (var i = 0; i < list.length; i += 1) { + var text = typeof list[i] === 'string' ? list[i] : ''; + for (var j = 0; j < text.length; j += 1) { + var code = text.charCodeAt(j); + h1 = Math.imul(h1 ^ code, 16777619) >>> 0; + h2 = Math.imul(h2 ^ code, 2246822519) >>> 0; + } + // A separator, so that ["ab", "c"] and ["a", "bc"] are two different sets. + h1 = Math.imul(h1 ^ 0x1f, 16777619) >>> 0; + h2 = Math.imul(h2 ^ 0x1f, 2246822519) >>> 0; + } + return (h1 >>> 0).toString(16).padStart(8, '0') + (h2 >>> 0).toString(16).padStart(8, '0'); + } + + /** + * The version tokens of the records on screen, in the order they are shown. + * + * The COUNT comes with them, because the service folds over ITS first N records and N is + * what says which of its records the fold is about. A page that is behind because the + * session grew has FEWER records, not different ones, so its first N are still the same N. + * + * @returns {Array} + */ + function displayedRecordVersions() { + var out = []; + var flat = Array.isArray(state.flat) ? state.flat : []; + for (var i = 0; i < flat.length; i++) { + var entry = flat[i]; + var record = entry && entry.record ? entry.record : entry; + out.push(record && typeof record.v === 'string' ? record.v : ''); + } + return out; + } + + /** + * WHICH SET the page's search answer is about. + * + * The session and the slice say WHICH BODY; the fingerprint says WHICH RECORDS of it. A body + * can change without changing its shape — a record's text rewritten under the page keeps every + * counter the same — so a comparison of counters cannot see that the records behind an answer + * are not the ones it was computed about. This is the same set the search request names, so + * the standing answer and the request cannot disagree about which records it covers. + * + * @param {Array} [versions] the versions to fold, when the caller already has them + * @returns {string} + */ + function displayedRecordSet(versions) { + var list = Array.isArray(versions) ? versions : displayedRecordVersions(); + return displayedSessionParam() + '#' + + (displayedGeneration() === null ? 'all' : String(displayedGeneration())) + + '#' + recordSetFingerprint(list); + } + + /** + * The set of records the page holds, as a request states it. + * + * The same question the search request asks, and for the same reason: the answer is about a + * SET of records, and a body can change without changing its shape — a record rewritten in + * place keeps every counter the same, so how many records the page has says nothing about + * whether its copies are still the service's. The versions do. + * + * @returns {{ count: number, fingerprint: string }} + */ + function heldRecordSet() { + var versions = displayedRecordVersions(); + return { count: versions.length, fingerprint: recordSetFingerprint(versions) }; + } + + /** + * The query a tail is asked with: the cursor, and the set the page holds. + * + * The cursor says how far the page got; the set says which records that is. A tail asked + * for without the set is a tail the service cannot answer without guessing: the records + * before the cursor are not in the tail, so the service would have no way of knowing + * whether the page's copies of them are still its own — and a record rewritten in place + * keeps its index and its id, so nothing else would say so either. + * + * @returns {string} + */ + function tailQuery() { + var held = heldRecordSet(); + return '&since=' + encodeURIComponent(String(state.since)) + + '&n=' + held.count + + '&fp=' + encodeURIComponent(held.fingerprint); + } + + /** + * The version of one record, as the record itself carries it. + * + * The record, and NOT the body on screen. A read that comes back 「this session is not + * cached」 reloads the body and asks again — and the body it reloads is a DIFFERENT one, so a + * version read off the screen at that moment belongs to the new body's record, and the retry + * would be asking about a different record than the one the reader opened. The record the + * reader opened is the one that carries the answer to 「which record is this」, so it is the + * one that carries the version. + * + * @param {Record} record + * @returns {string} the version, or '' when the record carries none + */ + function recordVersionOf(record) { + return record && typeof record.v === 'string' ? record.v : ''; + } + /** * The session list, on its own validator. * @@ -3990,6 +4309,7 @@

检视

if (sessionsInflight) sessionsInflight.abort(); var controller = new AbortController(); sessionsInflight = controller; + var seq = (sessionsSeq += 1); var key = 'limit=20'; var query = '?' + key; var headers = { Accept: 'application/json' }; @@ -4004,10 +4324,14 @@

检视

// Nothing moved since the copy already on screen. Keep every piece of state and // re-render nothing: the picker, its 「刚刚」 labels and its counts are all still true. if (response.status === 304) { + // The validator is written by the one writer below, and that runs after the + // sequence gate. A request that has been replaced must write nothing at all: a + // 304 that arrived late used to put its own older tag over the newer one, and + // the list on screen and the tag that describes it were then two different + // things -- so the next poll asked the server about the old list and kept the + // new one. var fresh = response.headers.get('etag'); - if (fresh) state.sessionsEtag = fresh; - state.sessionsEtagFor = key; - return { changed: false, body: null, etag: state.sessionsEtag }; + return { changed: false, body: null, etag: fresh || state.sessionsEtag }; } if (!response.ok) { // The status is what tells a dropped connection apart from a server that answered @@ -4019,14 +4343,26 @@

检视

return response .json() - .catch(function () { return null; }) + .catch(function (error) { + // An aborted request is not a broken answer. Swallowing it here reported a list + // that failed for a request that was simply replaced by a newer one. + if (error && error.name === 'AbortError') throw error; + return null; + }) .then(function (body) { if (!body || typeof body !== 'object') throw new Error('api'); return { changed: true, body: body, etag: response.headers.get('etag') || '' }; }); }) .then(function (result) { - sessionsInflight = null; + // The controller this request installed is the only one it may clear. A request that + // has been replaced must not take the next request's controller with it: that is how + // the next request stopped being able to abort this one, and an older answer ended + // up replacing a newer list. + if (sessionsInflight === controller) sessionsInflight = null; + // A late answer must not win. Every request carries a sequence number, and only the + // newest one may write -- the same rule the trajectory poll follows. + if (seq !== sessionsSeq) return false; // One writer for the list's validator, on the one path that carried a body. state.sessionsEtag = result.etag; state.sessionsEtagFor = key; @@ -4040,13 +4376,21 @@

检视

return false; } state.sessions = Array.isArray(result.body.sessions) ? result.body.sessions : []; + // The two lists are two views of one set: the service sends the main sessions and the + // runs of every conversation in the pool, and every run row names the main it hangs + // off. The page groups them by that name instead of keeping a hierarchy of its own. + state.children = Array.isArray(result.body.children) ? result.body.children : []; state.sessionsLoaded = true; renderSessionSelect(); + renderChildSelect(); if (cleared) renderBars(); return true; }) .catch(function (error) { - sessionsInflight = null; + if (sessionsInflight === controller) sessionsInflight = null; + // A request that has been replaced says nothing about the list on screen: the + // request that replaced it is the one that gets to report a failure. + if (seq !== sessionsSeq) return false; if (error && error.name === 'AbortError') return false; noteFresh('sessions', false, freshnessReason(error)); renderBars(); @@ -4072,8 +4416,12 @@

检视

// and the delta only ever arrives when the session really did grow. A manual refresh // drops the cursor as well as the validator: it is asking for the truth from the // server, and a tail is not that. + // + // The tail says which set the page holds as well, because the records before the + // cursor are not in the tail: the service can only answer one if it knows that the + // page's copies of those records are still its own. + (!manual && state.sinceFor === cacheKey && state.since > 0 - ? '&since=' + encodeURIComponent(String(state.since)) : ''); + ? tailQuery() : ''); var headers = { Accept: 'application/json' }; // A manual refresh must always reach the server: drop the validator so the // response cannot be revalidated into a 304 the user did not ask for. @@ -4092,10 +4440,13 @@

检视

// Nothing changed since the copy already on screen. Keep every piece of state // — no re-flatten, no re-render — and let the next poll do the same cheaply. if (response.status === 304) { + // The validator is written by the one writer below, and that runs after the + // sequence gate. A request that has been replaced must write nothing at all: a + // 304 that arrived late used to put its own older tag over the newer one, so the + // body on screen and the tag that described it were two different things — and + // the next poll asked the server about the old one and kept the new one. var fresh = response.headers.get('etag'); - if (fresh) state.etag = fresh; - state.etagFor = cacheKey; - return null; + return { changed: false, body: null, etag: fresh || state.etag }; } return response .json() @@ -4112,15 +4463,21 @@

检视

} if (!body || typeof body !== 'object') throw new Error('api'); var tag = response.headers.get('etag'); - state.etag = tag || ''; - state.etagFor = cacheKey; - return body; + return { changed: true, body: body, etag: tag || '' }; }); }) - .then(function (payload) { + .then(function (result) { + // A late answer must not win. Every request carries a sequence number, and only + // the newest one may write — the same rule the session list follows. if (seq !== refreshSeq) return; inflight = null; - if (payload === null) { + // One writer for the trajectory's validator, on the one path that carried an + // answer. A request that has been replaced writes neither a body nor a tag: the + // tag is what the next poll asks the server with, so a stale one leaves the page + // describing a body it is no longer showing. + state.etag = result.etag; + state.etagFor = cacheKey; + if (!result.changed) { // 304: the previous render is still accurate. Only clear a pending loading // state, because nothing on screen should have been replaced. // @@ -4133,17 +4490,26 @@

检视

state.loading = false; cleared = true; } - // Rendering on every 304 would rebuild the whole ledger every 8 seconds on an + // Rendering on every 304 would rebuild the whole ledger on every poll of an // idle session. Escaping the sticky banner must not cost that. if (cleared) render(); return; } - applyPayload(payload, manual); + applyPayload(result.body, manual); }) .catch(function (error) { if (seq !== refreshSeq) return; inflight = null; if (error && error.name === 'AbortError') return; + // The service will not answer a tail to a page whose copy is not the one it has, and + // it will not answer a tail asked for without saying which set the page holds. + // Either way the page's copy cannot be laid on top of what comes back, so the only + // answer left is the whole body — asked for again, without the cursor and without + // the validator that belonged to the copy that is gone. + if (isStaleCopyRefusal(error)) { + reloadWholeBody(); + return; + } handleTrajectoryError(error); }); } @@ -4167,6 +4533,45 @@

检视

* * @param {string} needle */ + /** + * An answer that is not about the body on screen. + * + * One place, because there are two ways to find that out, and they mean the same thing: a + * hit whose position holds a different record, and a service that says the body it searched + * is not this one. Both mean the answer cannot be installed, and both are answered the same + * way — the body is fetched once more and the same question is put again against it. + * + * Bounded, because a session that is being written to keeps moving: after a couple of tries + * the page stops chasing it, says that the answer does not apply, and the next poll's own + * re-run takes over. + * + * @returns {boolean} whether the page is going to ask again + */ + function searchAnswerDoesNotApply() { + if (state.searchRefresh < 2) { + state.searchRefresh += 1; + state.searchError = null; + state.searchPending = true; + // The whole body, not a tail: a delta carries the records after the cursor, and the + // record that moved is not one of them. + state.since = 0; + state.sinceFor = ''; + state.etag = ''; + state.etagFor = ''; + fetchTrajectory(false).then(function () { + runSearch(state.query); + }, function () { + state.searchPending = false; + state.searchError = 'search_version_mismatch'; + renderLedger(); + }); + return true; + } + state.searchError = 'search_version_mismatch'; + renderLedger(); + return false; + } + function runSearch(needle) { if (searchController) searchController.abort(); var asked = String(needle == null ? '' : needle).trim(); @@ -4175,17 +4580,46 @@

检视

state.searchPending = false; state.searchTotal = 0; state.searchTruncated = false; + state.searchRefresh = 0; + state.searchRefreshFor = ''; + // No question, no set. An answer about a set that is no longer on screen would filter + // rows by a question nobody is asking. + state.searchSet = ''; searchSeq += 1; return; } + // A new needle is a new question, and the counter that bounds the re-asking belongs to + // the question that started it. + if (state.searchRefreshFor !== asked) { + state.searchRefreshFor = asked; + state.searchRefresh = 0; + } var controller = new AbortController(); searchController = controller; var seq = (searchSeq += 1); state.searchPending = true; - var session = sessionParam(); + // The search is about the body on screen, so it asks about THAT session — the same + // question the raw and detail reads ask. Asking for the current session instead would + // answer with another session's records, and their indexes would then filter these rows. + // The slice is asked about the same way: the reader's selection and the body on screen + // are two different things, and a selection whose load failed leaves the old body up. + var session = displayedSessionParam(); + var generation = displayedGeneration(); + // WHICH BODY. The answer is about a SET of records, and the part of it that matters to + // the reader is the part it does not name: a record this body has and the other body does + // not match would be dimmed as if it did not contain the needle, with no error anywhere. + // So the request says which set it is about — how many records the page has, and a + // fingerprint of their versions — and the service answers only if it searched THAT set. + var versions = displayedRecordVersions(); + // The set this question is about, recorded before it is asked. The body on screen can be + // replaced while the answer is in flight, and a poll that finds the set has moved has to + // know that the answer it is waiting for is about the old one. + state.searchSet = displayedRecordSet(versions); var url = '/api/trajectory/search?session=' + session + '&q=' + encodeURIComponent(asked) - + (state.generation === null ? '' : '&generation=' + encodeURIComponent(String(state.generation))); + + (generation === null ? '' : '&generation=' + encodeURIComponent(String(generation))) + + '&n=' + versions.length + + '&fp=' + encodeURIComponent(recordSetFingerprint(versions)); fetch(url, { signal: controller.signal, headers: { Accept: 'application/json' } }) .then(function (response) { @@ -4206,9 +4640,78 @@

检视

if (seq !== searchSeq) return; searchController = null; state.searchPending = false; - state.searchHits = searchHitSet(body.hits); - state.searchTotal = Number.isInteger(body.total) ? body.total : 0; + // The answer has to be about the body on screen BEFORE its indexes may filter that + // body. Three things have to agree, and each of them answers a different question: + // + // - WHICH SESSION. The server names the session it answered for, and a body of + // another session's records would report a miss for a row that matches. + // - WHICH SLICE. The slice is compared against the body's own, not against the + // reader's selection: a selection whose load failed leaves the previous body on + // screen, and an answer about the selected slice would then filter rows that are + // not the ones on screen. + // - WHICH VERSION. The session and the slice say WHICH BODY; they say nothing about + // whether the record at a given index is still the same record. A transcript + // rewritten under a running page leaves the indexes alone and changes the records + // behind them, and an answer about the rewritten body would then light up a row + // whose text does not contain the needle at all. + var answered = body && body.session ? body.session : null; + var displayed = state.payload && state.payload.session ? state.payload.session : null; + // The two slices are compared as the page reads them: a number names one generation and + // nothing names the whole lineage. So a body that does not carry the field at all and a + // body that carries the whole lineage are the same question, and only a real difference + // between two real generations is a mismatch. + var answeredSlice = answered && typeof answered.generation === 'number' + ? answered.generation : null; + var displayedSlice = displayed && typeof displayed.generation === 'number' + ? displayed.generation : null; + if (!answered || !displayed || answered.id !== displayed.id + || answeredSlice !== displayedSlice) { + var mismatch = new Error('session_mismatch'); + mismatch.code = 'session_mismatch'; + throw mismatch; + } + // Every hit is checked against the body on screen, one by one, because a hit is a + // POSITION and a position only names a record when the version agrees. + // + // A hit whose position is on screen and whose version differs is about another version + // of that record: the whole answer is refused rather than partly installed, because + // the answer is about one body and the page is showing another. The body is then + // fetched once more and the same question is put again against the current body. + // + // A hit whose position is NOT on screen is about a record this body does not have — the + // session grew after the body was built. It cannot be shown, so it is not counted + // either: the number the reader sees has to be the number of rows that are lit up. + var byIndex = displayedRecordsByIndex(); + var hits = Array.isArray(body.hits) ? body.hits : []; + var applicable = []; + var wrongVersion = false; + for (var h = 0; h < hits.length; h += 1) { + var hit = hits[h]; + if (!hit || typeof hit.index !== 'number') continue; + var shown = byIndex.get(hit.index); + if (!shown) continue; + if (typeof shown.v === 'string' && shown.v !== '' && shown.v !== hit.v) { + wrongVersion = true; + break; + } + applicable.push(hit); + } + if (wrongVersion) { + // The whole answer is refused rather than partly installed, because the answer is + // about one body and the page is showing another. The body is then fetched once + // more and the same question is put again against the current body. + searchAnswerDoesNotApply(); + return; + } + state.searchHits = searchHitSet(applicable); state.searchTruncated = body.truncated === true; + // The count is the number of matches IN THE BODY ON SCREEN, so it is the number of rows + // the reader can see lit up. A truncated answer is the one exception: there the server's + // total is the real count and the rows on screen are only the first N of them, which + // the 「已截断」 mark says. + state.searchTotal = state.searchTruncated + ? (Number.isInteger(body.total) ? body.total : 0) + : applicable.length; // An answer that arrived retires the failure that preceded it. This line is here // because the two other places that clear the field are the two a READER triggers — // typing in the box and pressing clear — so nothing else could. A poll that re-runs the @@ -4217,6 +4720,7 @@

检视

// session: the label reads the error first, so the stale one outranked the fresh hit // set it was describing. state.searchError = null; + state.searchRefresh = 0; resetWindowMode(); renderLedger(); }) @@ -4225,6 +4729,13 @@

检视

searchController = null; state.searchPending = false; if (error && error.name === 'AbortError') return; + // The service refused because the body it searched is not the one on screen. That is + // the same thing a hit with the wrong version says, and it is answered the same way: + // fetch the body again and ask the question against it. + if (error && error.code === 'search_version_mismatch') { + searchAnswerDoesNotApply(); + return; + } // A search that could not be answered leaves the previous answer standing rather than // emptying the table: 「没搜到」 and 「没搜成」 are different, and only one of them is // true. The count line carries the reason. @@ -4381,6 +4892,38 @@

检视

return false; } + /** + * The copy on screen is not the one the service has: the whole body is asked for again. + * + * The cursor goes with it — a cursor into a copy that no longer exists — and so does the + * validator, which described that copy. What comes back is the whole body, which is the + * one answer that cannot be laid on top of a stale copy. + */ + function reloadWholeBody() { + state.since = null; + state.sinceFor = null; + state.loading = true; + render(); + fetchTrajectory(true); + } + + /** + * Whether the service's answer is a refusal to answer a tail to this page's copy. + * + * Two refusals, and both of them mean the same thing: the records the page holds are not + * the ones the service has — either because they have changed under it, or because the + * request never said which set it was holding. Neither is a failed poll: the request was + * answered, and the answer is that it cannot be answered with a tail. + * + * @param {any} error + * @returns {boolean} + */ + function isStaleCopyRefusal(error) { + if (!error || typeof error.code !== 'string') return false; + return error.code === 'trajectory_version_mismatch' + || error.code === 'trajectory_fingerprint_required'; + } + function applyPayload(body, manual) { // A delta is only ever laid onto a payload this page actually has. Without one — // first load, a session switch, a manual refresh — the whole payload is the answer. @@ -4389,11 +4932,7 @@

检视

// BEFORE the merge and returns early, because the alternative — merging it and noticing // afterwards — leaves records in the wrong turn with nothing to undo them. if (hasDelta && !deltaMatchesNumbering(state.payload, body)) { - state.since = null; - state.sinceFor = null; - state.loading = true; - render(); - fetchTrajectory(true); + reloadWholeBody(); return; } var mergeable = hasDelta; @@ -4429,7 +4968,20 @@

检视

// the old answer would show the reader rows that never matched. Re-running it here is the // whole of the rule; doing it at the reader would leave the poll path stale, which is the // same mistake the window pinning above used to make. - if (changed && state.query) runSearch(state.query); + // + // WHICH SET, and not only which shape. A body can change without changing its shape: a + // record's text rewritten under the page keeps every counter the same, so `changed` is + // false and the answer standing on screen is still about the old text — it would light up + // a row whose text does not contain the needle, and nothing anywhere would say so. The + // set the answer is about is the set of record versions it was computed against, which is + // the same set the request names; so the standing answer and the request cannot disagree + // about which records it covers. + // + // The set is only compared once a question has been asked (`state.searchSet` is empty + // until then), and a question that was never put has no answer to retire. + if (state.query && (changed || (state.searchSet && displayedRecordSet() !== state.searchSet))) { + runSearch(state.query); + } render(); if (manual) announce('已刷新，正在显示最新数据。'); } @@ -4453,39 +5005,97 @@

检视

/* The count is omitted whole when there is none, rather than printed as a dash. * - * `/api/sessions` reads the first 64 KB of the active `messages.jsonl` to build this list, so - * it used to send a message count that was really 「how many messages are in the first 64 KB」. - * On a session whose active file is 1 MB that is 10 against 23,642 — and this line put it in + * `/api/sessions` reads only a prefix of the active `messages.jsonl` to build this list, so + * it used to send a message count that was really 「how many messages are in that prefix」. + * When the prefix was a small part of the file, that count was wrong by orders of magnitude — and this line put it in * front of the reader as `（3 分钟前 · 10 条）`, with nothing to say it was a prefix. It now * sends no count when the prefix was not the whole file, so `— 条` here would be printing the * absence of a number in the shape of a number. Omitting the segment is the honest rendering * of 「这个数不存在」: the reader still has the label and the recency, which is what the picker * is for. tests/verify-list-cache.mjs pins the gate that produces the absence. */ - /* 「子会话」 says the structural fact and stops there. + /* The two picks, and what each one answers. * - * The parent is named by nobody, on purpose. A branch is always newer than the session that - * spawned it, so its parent has usually aged out of the most-recent-N window this list is - * built from — naming it would mean a lookup that fails exactly when a reader wants the - * answer most, and a lookup that fails quietly is how a picker ends up asserting something - * it cannot see. + * A run the runtime spawned from another session is a session like any other, and the runtime + * hangs it off its parent with one column. The page used to draw both in one dropdown, flat: + * most of the rows were runs of the one conversation being read, and nothing on the row said + * so. * - * It is a marker and not a filter: nothing here writes state, so there is no second opinion - * about which records are on screen that could disagree with the table. */ - var BRANCH_MARK = ' · 子会话'; - + * It then drew them inside the same dropdown, indented under their conversation. That made the + * hierarchy visible and made the dropdown long, so picking one conversation meant scrolling a + * list of titles to find it. + * + * It is two controls now, and the second is a cascade of the first: + * + * - 「会话」 answers 「which conversation」. It holds conversations and nothing else. + * - 「子代理」 answers 「which run inside it」. It holds the runs of the conversation the first + * pick names, and stays empty until there is one to name. + * + * Two controls is not two answers to one question: the questions are nested, and each has one. + * The layer is the container now, which says more than an indent or a word did — a row under + * 「子代理」 says what it is without saying it in words. + * + * A run is a session with its own transcript, and the parent transcript does not contain its + * turns. So picking a run shows that run's own trajectory, and the conversation pick moves to + * its parent: the two controls always agree about what is on screen. */ function sessionOptionLabel(session, now) { - var label = truncateText(session && session.label ? session.label : '', 42); - var when = formatAgo(session ? session.lastActiveAt : null, now); - var count = session ? session.messageCount : null; - var tail = isCount(count) ? '（' + when + ' · ' + formatInt(count) + ' 条）' : '（' + when + '）'; - var branch = session && session.sessionParentId ? BRANCH_MARK : ''; - if (!label) return '未命名会话 ' + tail + branch; - return label + ' ' + tail + branch; + var parts = sessionLabelParts(session, now); + if (!parts.label) return '未命名会话 ' + parts.tail + parts.tag; + return parts.label + ' ' + parts.tail + parts.tag; + } + + /** + * The sub-agents of each conversation, grouped by the parent the service named. + * + * One pass over the flat list the service sends, and the order inside each group is the + * order it arrived in — most recently active first, which is the service's ordering and the + * only one the picker promises. + * + * @returns {Object} + */ + function childrenByParent() { + // A map with no prototype. The parent id is data that crossed the wire from the + // runtime database, and the names a plain object inherits -- '__proto__', + // 'constructor' -- are truthy and are not arrays, so a plain object reads one of them + // as a group and then throws on it, and the picker stops updating after an otherwise + // successful list. + var grouped = Object.create(null); + var all = state.children || []; + for (var i = 0; i < all.length; i += 1) { + var child = all[i]; + if (!child || !child.id || !child.sessionParentId) continue; + var bucket = grouped[child.sessionParentId]; + if (bucket) bucket.push(child); + else grouped[child.sessionParentId] = [child]; + } + return grouped; + } + + /** + * The conversation the session on screen belongs to. + * + * A run belongs to the conversation that spawned it; a conversation belongs to itself. That + * is the only thing the two picks have to agree about, and it is derived rather than stored: + * the parent id comes down with the row, so there is no second copy of the mapping to keep + * in step with the service. + * + * @returns {string} + */ + function ownerSessionId() { + var id = state.sessionId || ''; + if (!id) return ''; + var all = state.children || []; + for (var i = 0; i < all.length; i += 1) { + var child = all[i]; + if (child && child.id === id && child.sessionParentId) return child.sessionParentId; + } + return id; } function renderSessionSelect() { var select = dom.sessionSelect; - var previous = state.followLatest ? 'latest' : state.sessionId || ''; + // The conversation pick names the conversation, not the run: picking a run leaves this + // control on the run's parent, so the two controls never disagree about what is on screen. + var previous = state.followLatest ? 'latest' : ownerSessionId(); clear(select); var followOption = el('option', null, '跟随最新（自动切换）'); @@ -4515,6 +5125,50 @@

检视

if (select.selectedIndex < 0) select.value = 'latest'; } + /** + * The run pick: the runs of the conversation on screen, and nothing else. + * + * Empty while the page follows the newest conversation — that list changes under the reader, + * and a run picked out of it would be a run of a conversation that is no longer on screen. + * Empty with a conversation picked and no run picked, which is the default; the first option + * says so in words. + */ + function renderChildSelect() { + var select = dom.childSelect; + clear(select); + var owner = state.followLatest ? '' : ownerSessionId(); + + if (!owner) { + var idle = el('option', null, '（跟随最新）'); + idle.value = ''; + select.appendChild(idle); + select.disabled = true; + select.value = ''; + return; + } + + select.disabled = false; + var self = el('option', null, '（主会话本身）'); + self.value = ''; + select.appendChild(self); + + var now = Date.now(); + var kids = childrenByParent()[owner] || []; + for (var j = 0; j < kids.length; j += 1) { + var option = el('option', null, sessionOptionLabel(kids[j], now)); + option.value = kids[j].id; + select.appendChild(option); + } + // The run on screen, when it is a run of this conversation; otherwise the default, which + // is the conversation itself. + var current = state.sessionId || ''; + var isRun = false; + for (var k = 0; k < kids.length; k += 1) { + if (kids[k].id === current) isRun = true; + } + select.value = isRun ? current : ''; + } + /* ---------------- rendering: KPI ---------------- */ /** @@ -4641,7 +5295,7 @@

检视

return wrap; } - /** What the 对话轮次 line cannot say by looking at it. Kept to six lines: it opens under the + /** What the 对话轮次 line cannot say by looking at it. Kept to a few lines: it opens under the * card and covers the kind chips, so every extra line hides the thing it points at. */ function turnTipLines(stats) { var total = isCount(stats.turnsWithPrompt) ? stats.turnsWithPrompt : 0; @@ -4663,7 +5317,7 @@

检视

* * The session total deliberately does NOT live here. Putting 「共 84 轮」 next to a table * that starts at 第 82 轮 invited exactly the reading "these should be equal" — and they - * are not, because the default window holds only the newest 200 records and one of those + * are not, because the default window holds only the newest PAGE_STEP records and one of those * turns alone holds 183 of them. The total is still on screen, on the window bar beside * how many records are not loaded, which is the question it actually answers. */ @@ -6040,7 +6694,7 @@

检视

* Whether the record the inspector is showing still belongs to the table. * * It used to ask `state.flat` — the whole session, unfiltered — so a record stayed selected - * after every filter had already removed it: the ledger read 已显示 2 条, the inspector still + * after every filter had already removed it: the ledger read 已显示 N 条, the inspector still * showed the row the reader had picked, and no row anywhere carried the selection mark. The * list handed in here is the filter result the table is narrowed from, not the session, so a * record this accepts is by construction one the filters kept. @@ -6404,8 +7058,8 @@

检视

* Anything that changes what the numbers mean — a filter, a new session, a fresh search, * a brush, a generation pin — resets to the default. Resetting is not about shrinking the * window: it is about discarding a window that was chosen for a set of records which no - * longer exists. A reader who was looking at only the newest 200 and then brushes an - * older range must not be left with a 200-row cap over a range holding thousands. + * longer exists. A reader who was looking at only the newest PAGE_STEP records and then brushes an + * older range must not be left with the default window's cap over a range holding thousands. * * @param {'latest'|'all'} mode */ @@ -6431,7 +7085,7 @@

检视

* What happens to the window when a poll brings new records. * * 全部 must not re-pin, or the first record to arrive after the click would cancel it - * and the reader would be thrown back to 200 records without ever being told why. + * and the reader would be thrown back to the default window without ever being told why. * * 「向上加载更早的 N 条」 must not re-pin either, and that is the older of the two. * `limit` is the reader's own choice, not a cache: at exactly PAGE_STEP the window is @@ -6439,14 +7093,14 @@

检视

* reader pressed a button asking for more, and a poll that resets the limit takes that * back without saying so. The button re-rendered and appeared to do nothing — the only * reason it was not noticed is that this session grows by a few records per second, so - * the count stayed near 200 either way. Growing the session hid a dead control. + * the count stayed near PAGE_STEP either way. Growing the session hid a dead control. * * @returns {number} the limit now in force */ function rePinWindow() { if (state.allRecords) return state.limit; // A time range is a fixed stretch of the session, so the window follows it rather - // than the tail. Re-pinning would slide the whole range forward every 2.5s, and the + // than the tail. Re-pinning would slide the whole range forward on every poll, and the // records that scrolled off the top would be records the reader had scoped to. if (hasTimeRange()) return state.limit; if (state.limit > PAGE_STEP) return state.limit; @@ -6454,13 +7108,594 @@

检视

return state.limit; } + /* ---------------- ledger group lifecycle ---------------- */ + + /** + * How long a group may keep its rows after the reader has left it. + * + * Long enough that scrolling back and forth over the same stretch does not rebuild it + * every time, short enough that the rows of a session the reader has moved past are not + * held for the rest of the session. + */ + var LEDGER_IDLE_MS = 15000; + + /** + * How many groups may hold their rows at once. + * + * The cap is what makes the memory bounded rather than merely reduced. A reader who + * scrolls through three hundred groups in a minute has been on all of them, and the idle + * timeout alone would not release them fast enough to stop it. Past the cap the oldest one + * goes, which is the order the reader would choose. + */ + var LEDGER_LIVE_GROUPS = 6; + + /** + * What is known about each turn group: whether it is holding its rows, when the reader was + * last on it, and how tall it was. + * + * Keyed by the turn's key, so it survives the poll that rebuilds the table. The DOM is + * thrown away and rebuilt on every poll; this is what carries the reader's + * place in it across that, and it is why the reader does not have to start again at the top + * of the session on every tick. + * + * @type {Map} + */ + var ledgerState = new Map(); + + /** + * The group each section was built from. + * + * A section that scrolls into view has to be filled, and filling it needs the records it + * stands for. The table is rebuilt on every poll, so this is rebuilt with it — a section + * from the previous table is not in here any more, and filling it would put rows into a + * table that no longer exists. + * + * @type {Map} + */ + var ledgerContent = new Map(); + + /** + * The row heights the last build calibrated: the head, a row and a band. + * + * The build measures them off the first batch, which is the only part of the table that + * has been laid out for real. The visit path needs them as well: it has to bring the + * height a released group keeps up to date with what the group holds now, and a group + * that has gained a row since it was measured is taller by a row's worth of height. + * + * One writer: the build, right after it has measured the batch. The visit path reads it + * and never writes it. + * + * @type {{head: number, row: number, band: number}|null} + */ + var ledgerCalibration = null; + + /** + * The observer that fills a group when the reader reaches it. + * + * One observer for the whole table: the sections are rebuilt on every poll, so an observer + * per section would have to be rebuilt with them. + */ + var ledgerObserver = null; + + /** + * The state of one group, created on first sight. + * + * `heightRows` and `heightBands` say what the measured height was a measurement OF. A + * height on its own describes a past: the group can hold a different number of rows by + * the time the height is read again, and without the two counts the difference cannot be + * charged to anything. + * + * @param {string} key + * @returns {{live: boolean, lastSeen: number, height: number, heightRows: number, heightBands: number}} + */ + function ledgerStateOf(key) { + var state = ledgerState.get(key); + if (!state) { + state = { live: false, lastSeen: 0, height: 0, heightRows: 0, heightBands: 0 }; + ledgerState.set(key, state); + } + return state; + } + + /** + * Whether a group should be holding its rows right now. + * + * Three things make a group live: the reader is on it, the reader was on it recently and + * the idle timeout has not passed, and it is the first group of the window — which is where + * the reader arrives, and the only source of the heights every other group is predicted + * from. + * + * @param {string} key + * @param {boolean} isFirst + * @returns {boolean} + */ + function ledgerHoldsRows(key, isFirst) { + if (isFirst) return true; + var state = ledgerState.get(key); + if (!state || !state.live) return false; + return Date.now() - state.lastSeen < LEDGER_IDLE_MS; + } + + /** + * Decide which groups give up their rows. + * + * Three rules, and the order matters. A group the reader has left goes when the idle + * timeout has passed; when more groups are live than the cap allows, the oldest goes + * first; and the groups the caller is holding on to are none of the reader's visits at + * all, so they are counted rather than timed. + * + * That last one is what keeps the bound about the rows on screen. The groups the caller + * holds on to hold their rows whatever the reader is doing — the window's first group is + * where the reader arrives and every other group's height is predicted from it — so a + * bound on the reader's groups alone would leave one group holding its rows outside the + * count. Counting them here makes the two sets the same set: the groups that are live + * are the groups that hold their rows. + * + * One decision and two callers, and that is the point of it. The bound is about what the + * reader is holding at this moment, not only about what a rebuild leaves behind: a reader + * who walks through more groups than the cap allows between two rebuilds is holding all + * of them until the next poll comes round. So the rebuild and the visit both come through + * here, and they cannot disagree about which groups go — the same order, the same count. + * + * They do not hold on to the same groups, and the difference is the point. The rebuild + * holds on to the window's first group. The visit holds on to that group as well, and to + * the group the reader has just reached: a visit that protected only the group the + * reader had just reached would let the window's first group go — the idle timeout + * would take it, or the cap would reach for it as the oldest one there is — and the + * group every height is predicted from would lose its rows while the reader was on the + * page. + * + * This decides. The rows are taken away by the build, which writes the released groups + * as it writes the table, and by the visit, which takes the rows out of a table that is + * already on screen. + * + * @param {string[]} keepKeys the groups that hold their rows whatever else happens: the + * window's first group, and on a visit the group the reader has just reached as well. + * The group the reader has just reached is the last of them, so that a visit in the same + * millisecond as another one cannot be put at the head of the queue. + * @returns {string[]} the groups that have to give up their rows + */ + function releaseStaleGroups(keepKeys) { + var now = Date.now(); + var live = []; + var doomed = []; + var kept = []; + var keptIndex = Object.create(null); + var keys = Array.isArray(keepKeys) ? keepKeys : [keepKeys]; + for (var k = 0; k < keys.length; k += 1) { + if (keys[k] === null || keys[k] === undefined) continue; + if (keptIndex[keys[k]]) continue; + keptIndex[keys[k]] = true; + kept.push(keys[k]); + } + ledgerState.forEach(function (state, key) { + if (keptIndex[key]) return; + if (!state.live) return; + if (now - state.lastSeen >= LEDGER_IDLE_MS) { + state.live = false; + doomed.push(key); + return; + } + live.push({ key: key, at: state.lastSeen }); + }); + // The groups the caller is holding on to are counted with the current time, so they are + // never the oldest ones the cap reaches for. Their leases are not consulted: they do + // not have one. + // + // They are counted after the others, so that a visit in the same millisecond as another + // one — a tie the clock cannot break — cannot put the group the reader has just reached + // at the head of the queue. The group the reader has just reached is the last of them, + // and the cap must not take it. + for (var j = 0; j < kept.length; j += 1) { + ledgerStateOf(kept[j]).live = true; + live.push({ key: kept[j], at: now }); + } + if (live.length <= LEDGER_LIVE_GROUPS) return doomed; + live.sort(function (a, b) { return a.at - b.at; }); + for (var i = 0; i < live.length - LEDGER_LIVE_GROUPS; i += 1) { + var state = ledgerState.get(live[i].key); + if (!state) continue; + state.live = false; + doomed.push(live[i].key); + } + return doomed; + } + + /** + * The key of the window's first group. + * + * The window's first group is the first section of the table: the build writes the groups + * in the order of the window, so the first section is the group the reader arrives at and + * the one every other group's height is predicted from. + * + * It is read out of the table rather than remembered in a variable of its own. A variable + * would be a second copy of the same fact, and the two copies would have to be kept in + * step; the table has one writer — the build — and its first section is the window's + * first group by construction. + * + * @returns {string|null} null when the table has no group in it + */ + function ledgerFirstGroupKey() { + var sections = dom.ledger.querySelectorAll(':scope > section.turn-group'); + if (!sections.length) return null; + var content = ledgerContent.get(sections[0]); + return content ? content.key : null; + } + + /** + * The groups that have to give up their rows before a rebuild writes the table. + * + * The rebuild and the visit make the same decision, so they come through the same + * function; this is the rebuild's half of it, and it is a function of its own so that the + * two halves cannot drift apart: the rebuild holds on to the window's first group, and + * the visit holds on to that group and to the group the reader has just reached. + * + * @param {Array<{turn: any}>} groups the window's groups, in order + */ + function releaseStaleGroupsBeforeRebuild(groups) { + if (!groups.length) return; + releaseStaleGroups([turnKey(groups[0].turn)]); + } + + /** + * Put a group's rows back into it. + * + * The group is in view, so this is what the reader is about to read, and the rows go in + * before the next paint — so there is no frame in which the group is empty. + * + * The rows go in once. A group that is already holding them is not built a second time, + * and the second copy of every row is the whole cost of getting it wrong. + * + * @param {HTMLElement} section + */ + function fillLedgerGroup(section) { + var group = ledgerContent.get(section); + if (!group) return; + var state = ledgerStateOf(group.key); + state.live = true; + state.lastSeen = Date.now(); + var alreadyLive = section.getAttribute('data-rows') === 'live'; + if (!alreadyLive) { + var list = section.querySelector('ul.row-list'); + if (!list) return; + fillGroupRows(list, group); + section.setAttribute('data-rows', 'live'); + section.classList.remove('is-released'); + section.style.minHeight = ''; + } + // The height the group really has, taken on both paths — see measureLedgerGroup(). + // A group that is already holding its rows is measured again when its LAYOUT has + // changed, and the test for that is the layout and not the content: the rows and the + // bands can be exactly the same and the height a different one — the window resized, + // the text scale turned up, the group's own chrome taller. The height that was + // measured then describes a layout that is no longer there, and the height a released + // group keeps is that height unchanged, because the counts it was measured at have + // not changed either — so the group stands shorter than the content it is holding, + // and the document under the reader comes up short by exactly that much. + // + // The group the reader is on is rendered, so the height it has now is its layout and + // not the prediction an off-screen group is sized by; the off-screen groups are the + // calibration-priced path's business, and this does not touch them. + if (!alreadyLive || !state.height || ledgerLayoutChanged(section, state)) measureLedgerGroup(section, group); + // The cap, on the visit path as well as on the rebuild: the number of groups holding + // their rows is bounded at any moment, not only right after a rebuild. + // + // Two groups are held on to: the group the reader has just reached, and the window's + // first group, which holds its rows whatever the reader is doing — it is where the + // reader arrives and the one every other group's height is predicted from. The group + // the reader has just reached is the last of the two, so that a tie the clock cannot + // break cannot put it at the head of the queue. + releaseLedgerRows(releaseStaleGroups([ledgerFirstGroupKey(), group.key])); + } + + /** + * Record how tall a group really is, and what it was holding when it was measured. + * + * The measurement belongs to the observer, because the observer is what sees the group + * laid out for real. The build never measures: a group it holds has no height of its own + * until somebody looks at it, and a build that stamped one would be writing down a + * height it never measured. + * + * Both paths record it. The group the build made live — the first group of the window is + * always live — is visited by the observer like any other, and it is laid out for real at + * that moment. Without a measurement here it keeps no height of its own, and when the + * window widens and the group is released it falls back on the prediction: the document + * under the reader loses the height the group really had, over a group whose rows and + * bands did not change by one. + * + * The counts go with the height, because a height is a measurement OF something. The group + * is keyed by its turn, and the turn keeps its key while its content changes — a late + * reply, a band from a compaction — so a height that is not read together with its counts + * describes a content that is no longer there. + * + * The two counts are rows and bands, and they are counted apart. A band is an entry as + * well as a row, so a count of entries would charge every band once as a row and once as + * a band — the band at 36 px and 38 px would be charged 74 px, and the box would stand + * taller than the content it is holding. + * + * @param {HTMLElement} section + * @param {{key: string, entries: any[], bands?: number}} group + */ + function measureLedgerGroup(section, group) { + var state = ledgerStateOf(group.key); + // The height it really has, now that it is laid out for real. A group the browser has + // skipped measures by its own contain-intrinsic-size, so this reads the prediction back + // rather than the truth — which is why it is the remembered height and not the only one. + sizeBuiltGroup(section); + state.height = Math.round(section.getBoundingClientRect().height); + var measuredBands = group.bands || 0; + state.heightRows = group.entries.length - measuredBands; + state.heightBands = measuredBands; + } + + /** + * Whether a group's layout is not the one its remembered height was measured at. + * + * The remembered height is a measurement of a layout that was there then, and a layout + * can change without the content changing: the window is resized, the text scale is + * turned up, the group's own chrome becomes taller. The rows and the bands can be + * exactly the same and the height a different one — and the height a released group + * keeps is the remembered one unchanged, because the counts it was measured at have not + * changed either. So the group stands shorter than the content it is holding, and the + * document under the reader comes up short by exactly that much. + * + * So the test is the height itself: the group has a height of its own now, and it is + * not the one that was remembered. + * + * A group with no remembered height has no layout to compare against, and this says + * nothing about it: the caller has a rule of its own for that case, and it is the one + * that has to stay — the group the build made live has to record the height it was + * measured at, because the build never measures. + * + * @param {HTMLElement} section + * @param {{height: number}} state + * @returns {boolean} + */ + function ledgerLayoutChanged(section, state) { + if (!state.height) return false; + return Math.round(section.getBoundingClientRect().height) !== state.height; + } + + var ledgerMeasureFrame = null; + + /** + * Measure the groups that are on screen again, after the window has changed shape. + * + * WHY: a height is a measurement of a layout, and a resize changes the layout of every + * group that is rendered without changing a row or a band in it. The measurement is the + * observer's, and the observer is not told that the window changed shape — so a group + * the reader is looking at keeps the height it was measured at, and that height + * describes the layout before the resize, until the reader scrolls away from it and + * back. And when the cap releases it, the height it reserves is that stale number, so + * the document under the reader comes up short by exactly the difference. + * + * The measurement is deferred to the next frame, because the layout a resize brought + * about is not there yet when the event arrives. + * + * What it does not claim: a group that was off screen when the layout changed, and is + * never visited again, keeps the height it was last really measured at. Off screen the + * browser answers with the group's own `contain-intrinsic-size`, and that number is a + * prediction — a prediction written down as a measurement is a guess the next rebuild + * reads back as a fact. So a group that reads back the size it was given is left alone + * here, and the observer's visit is what measures a group that happens to be laid out at + * exactly that height — the same limit the build path has. + */ + function refreshLedgerMeasurements() { + if (ledgerMeasureFrame !== null) return; + ledgerMeasureFrame = window.requestAnimationFrame(function () { + ledgerMeasureFrame = null; + var sections = dom.ledger.querySelectorAll(':scope > section.turn-group'); + for (var i = 0; i < sections.length; i += 1) { + var section = sections[i]; + // Only the groups that are holding their rows. A group that has given them up is + // sized by the prediction, and the height it keeps is the one it was measured at. + if (section.getAttribute('data-rows') !== 'live') continue; + var group = ledgerContent.get(section); + if (!group) continue; + // The size the browser answers with while a group is off screen is the size + // written on it — so a group that reads back exactly the number it was given MAY + // be being sized rather than laid out, and what it reads is then a prediction. + // It is not proof of that: the first batch's own size writer puts the group's REAL + // height on it, so a group that is laid out reads back its own height too. What + // keeps the rule true is the observer: a group the reader is on is measured again + // by the observer's visit, and the rebuild is what re-registers that observer. A + // group that is neither visited nor observed keeps the height it was measured at, + // and that is the same limit the build path has. + var sizes = String(section.style.containIntrinsicSize || '').match(/(-?[0-9.]+)px/g); + if (sizes) { + var predicted = Math.round(parseFloat(sizes[sizes.length - 1])); + if (Math.round(section.getBoundingClientRect().height) === predicted) continue; + } + // The rows are not built here: a group that is holding them is holding them, and + // the second copy of every row is the whole cost of getting it wrong. + measureLedgerGroup(section, group); + } + }); + } + + /** + * Take the rows out of the groups that have to give up their rows. + * + * The rebuild writes the released groups as it builds the table, so there is nothing to do + * there. A visit happens between two rebuilds, and the groups it releases are in a table + * that is already on screen: their rows have to go now, or the reader is holding more + * rows than the cap allows until the next poll comes round. + * + * The height is kept. The rows are the DOM; the height is the reader's place in a session + * they have scrolled through, and a group that has given up its rows has to keep it or + * the document shrinks under them. + * + * @param {string[]} keys the groups that have to give up their rows + */ + function releaseLedgerRows(keys) { + if (!keys.length) return; + var sections = dom.ledger.querySelectorAll(':scope > section.turn-group'); + for (var i = 0; i < sections.length; i += 1) { + var section = sections[i]; + var content = ledgerContent.get(section); + if (!content) continue; + if (keys.indexOf(content.key) < 0) continue; + var bands = content.bands || 0; + var rows = content.entries.length - bands; + // The height the group keeps is the one it was measured at. It is read before the + // rows go, because after them the box is the group's own head and nothing else. + var measured = Math.round(section.getBoundingClientRect().height); + var list = section.querySelector('ul.row-list'); + if (list) clear(list); + section.setAttribute('data-rows', 'released'); + section.classList.add('is-released'); + // The measured height, brought up to date with what the group holds now; the height + // read above otherwise. The difference is priced at the row heights the last build + // calibrated, which is the same arithmetic the build uses. + // + // The height read above cannot be used on its own: a group the browser has skipped + // measures by its own contain-intrinsic-size, so for the groups that are off screen + // it is the prediction and not a measurement — and writing it would replace the + // height the group was really measured at with a guess. The calibration is what + // prices the difference, and it is what the build prices it with. + holdLedgerHeight(section, heldLedgerHeight(content.key, rows, bands, measured, ledgerCalibration)); + } + } + + /** + * Watch the groups, so a group the reader scrolls to gets its rows. + * + * The observer is disconnected and rebuilt on every poll, because the sections are: a + * section from the previous table is not in `ledgerContent` any more. + */ + function watchLedgerGroups() { + if (!ledgerObserver) { + ledgerObserver = new IntersectionObserver(function (entries) { + for (var i = 0; i < entries.length; i += 1) { + if (!entries[i].isIntersecting) continue; + fillLedgerGroup(entries[i].target); + } + }); + } else { + ledgerObserver.disconnect(); + } + var sections = dom.ledger.querySelectorAll(':scope > section.turn-group'); + for (var i = 0; i < sections.length; i += 1) ledgerObserver.observe(sections[i]); + } + + /** + * Hold a group's height while it is not holding its rows. + * + * `contain-intrinsic-size` sizes a group the browser has skipped, and `min-height` sizes + * one it is rendering. A group near the viewport is rendered, so it needs the second, and + * without it the document under the reader loses the height of everything they have + * scrolled past — measured at 79,288 px for the groups around the viewport, and 257,378 px + * more for the viewport's own group. + * + * @param {HTMLElement} section + * @param {number} height + */ + function holdLedgerHeight(section, height) { + if (!(height > 0)) return; + section.style.minHeight = Math.round(height) + 'px'; + } + + /** + * The height a released group should keep, brought up to date with what it holds now. + * + * The remembered height is a measurement of a past. It was taken when the group was + * holding a particular number of rows and bands, and the group can gain one of each + * after the reader has walked away from it — a late reply, a band from a compaction — + * because a group is keyed by its turn, and the turn keeps its key while its content + * changes. A height kept unchanged then describes content that is no longer there, and + * the document under the reader comes up short by exactly the rows that were added. + * + * So the difference between what the group holds now and what it held when it was + * measured is charged at the rates the prediction uses: a row's worth of height for + * every row added, a band's worth for every band. The two counts are kept apart, so a + * band is charged as a band and not as a row as well — the two heights are close (36 px + * and 38 px, measured on a real session), and a band charged twice would leave the box + * a whole row taller than the content it is holding. What is left of the measurement is + * the part that belongs to the group rather than to its content — its own header, its + * borders — and that is the part the prediction cannot know, because the prediction is + * priced from medians and this group's chrome is its own. + * + * The difference is local: the measured height is kept as it was, and only the rows + * that changed are priced. That is the whole difference between this and falling back + * on the prediction, which would price the whole group again — and a group's real + * height runs to several times the median one. + * + * The measurement is the observer's: it is taken when the group is really laid out. + * This reads it and never writes it, because a build that stamped one would be writing + * down a height it never measured. + * + * What this does not claim: the difference is priced at the calibrated heights, which + * are medians, so a group whose own rows are taller than the median one has its + * difference undercounted. The height is brought up to date; it is not a measurement of + * what the group holds now. + * + * @param {string} key + * @param {number} rows how many rows the group holds now; the compaction bands are + * counted apart and are not rows + * @param {number} bands how many compaction bands the group holds now + * @param {number} predicted the height the shape of the group predicts + * @param {{head: number, row: number, band: number}|null} heights the calibrated heights + * @returns {number} + */ + function heldLedgerHeight(key, rows, bands, predicted, heights) { + var state = ledgerState.get(key); + if (!state || !(state.height > 0)) return predicted; + // A height measured without knowing what was in the group cannot be brought up to + // date: there is no difference to charge. + if (typeof state.heightRows !== 'number') return predicted; + // No calibration yet, so a difference in rows cannot be turned into pixels. The + // measured height is stale rather than right, and a stale height is better than none: + // the first batch is re-sized from the calibration as soon as there is one. + if (!heights) return state.height; + var addedRows = rows - state.heightRows; + var addedBands = bands - state.heightBands; + if (!addedRows && !addedBands) return state.height; + var held = state.height + addedRows * heights.row + addedBands * heights.band; + return held > 0 ? held : predicted; + } + + /** + * Build a group's rows into its list. + * + * The rows of a group are built in one place, and the table has two writers for them: the + * build, which makes the groups the reader is on, and the observer, which fills a group the + * reader has just scrolled to. A second copy of this loop would be two definitions of what + * a row is, and the page has already paid for that once. + * + * A `null` list counts the compaction bands without building anything, which is what a + * group that is not holding its rows needs: its height is predicted from how many bands it + * has, and a prediction that ignored them would be short by every band in it. + * + * @param {HTMLElement|null} list + * @param {{turn: any, entries: any[]}} group + * @returns {number} how many of the entries are compaction bands + */ + function fillGroupRows(list, group) { + var rails = list ? railStates(group.entries) : null; + var bands = 0; + for (var j = 0; j < group.entries.length; j += 1) { + var isBand = group.entries[j].record.kind === 'compaction'; + if (isBand) bands += 1; + if (!list) continue; + var item = el('li'); + if (isBand) { + item.className = 'compaction-cell'; + item.appendChild(buildCompactionBand(group.entries[j])); + } else { + item.appendChild(buildRow(group.entries[j], rails[j])); + } + list.appendChild(item); + } + return bands; + } + /* ---------------- batched row build ---------------- */ /** * Turn groups built per frame once a window is too large for one. * - * Measured, not chosen. On this session at 18,924 rows across 150 turns, 25 groups is - * ~3,150 rows and ~145 ms of row construction — the largest slice of a frame this loop can + * Measured, not chosen: a batch of LEDGER_BATCH_GROUPS is + * the largest slice of a frame this loop can * take before the page stops answering the pointer. * * The alternative was measured and rejected rather than untried: skipping layout for the @@ -6497,7 +7732,7 @@

检视

/** A poll tick that arrived while the ledger was still being built. */ var pollDeferred = false; - /** Whether a batched build is running. One tick every 2.5 s asks this before it redraws. */ + /** Whether a batched build is running. One tick on every poll asks this before it redraws. */ function ledgerBuilding() { return ledgerProgress !== null; } @@ -6578,6 +7813,10 @@

检视

only fixes the first build after a page load. This line is what fixes the rest. */ ledger.classList.remove('is-skipping-offscreen'); clear(ledger); + // The sections the last build made are gone with them, and the map that says which group + // each one stands for has to go with them: a section that is no longer in the table must + // not be filled when the observer reports it as in view. + ledgerContent.clear(); if (state.loading) { ledger.appendChild(buildSkeleton()); @@ -6634,7 +7873,7 @@

检视

// It said: what the reader can see is the window, not the filter result. That was written // to fix a real thing: this call used to RAISE `state.limit` to reach a selection // below the window, which made the window a thing the inspector could widen, and left the - // promise 「已显示 200 条」 true at the moment the reader pressed 显示最新 200 条 and false a + // promise 「已显示 N 条」 true at the moment the reader pressed 显示最新 N 条 and false a // second later. **That part is still true and the prohibition still stands** — the window // never widens to reach a selection, and it still does not below. // @@ -6643,7 +7882,7 @@

检视

// record slides the whole tail forward by one row. So 「not in the window」 was measuring // the projection's motion, not the reader's attention. Measured on a real session (32,386 // records, 216 turns, the reader touching nothing): a record selected at position p of the - // 200-row tail survived exactly `200 - p` new records, and p=0 died after ONE of them, + // tail survived exactly `PAGE_STEP - p` new records, and p=0 died after ONE of them, // inside a single poll. On a session that is still being written, that is every selection // that is not already the newest — which is why it read as 「过一会检视会消失」. // @@ -6723,6 +7962,12 @@

检视

var buildTotal = 0; for (var b = 0; b < groups.length; b += 1) buildTotal += groups[b].entries.length; + // The groups the reader has left give up their rows, and the ones past the cap give them + // up oldest first. The same decision the visit path makes, so the two cannot disagree + // about which groups go; applied by this build, which writes the released groups as it + // writes the table. + releaseStaleGroupsBeforeRebuild(groups); + function buildStep() { // A newer renderLedger() owns the ledger now. Stop without touching it: the rows half // built here belong to a table that no longer exists, and the exit that replaced it has @@ -6738,18 +7983,29 @@

检视

var list = el('ul', 'row-list'); // Every entry that survived the filters becomes a row. There is no second pass // that hides anything, so what is on screen is exactly what was selected. - var rails = railStates(group.entries); - var bands = 0; - for (var j = 0; j < group.entries.length; j += 1) { - var item = el('li'); - if (group.entries[j].record.kind === 'compaction') { - item.className = 'compaction-cell'; - item.appendChild(buildCompactionBand(group.entries[j])); - bands += 1; - } else { - item.appendChild(buildRow(group.entries[j], rails[j])); - } - list.appendChild(item); + // + // The rows are built for the groups the reader is on and released for the rest. What + // the reader gives up is the DOM, and the DOM is where the cost lives: the page pays + // for the rows of the groups the reader is on and nothing else. How many groups are + // live changes as the reader moves through the conversation, so there is no fixed + // count of held rows to quote here. + // + // What the reader does not give up is the height — a released group keeps the box it + // had, brought up to date with what it holds now, so the document under them does not + // change size when they scroll. + var key = turnKey(group.turn); + // The rebuild is not a visit. The poll runs on a timer whether the reader + // is there or not, so a build that stamped the lease would renew it forever: the + // idle timeout could never pass, and the group the reader had left would keep its + // row DOM for the rest of the session. The lease belongs to the observer, which + // only speaks when the reader is actually at the group. + var holds = ledgerHoldsRows(key, build.at === 1); + var bands = fillGroupRows(holds ? list : null, group); + if (holds) { + section.setAttribute('data-rows', 'live'); + } else { + section.setAttribute('data-rows', 'released'); + section.classList.add('is-released'); } section.appendChild(list); build.rows += group.entries.length; @@ -6759,8 +8015,20 @@

检视

// how a scrollbar ends up shorter than the table it is supposed to describe. On the // first batch build.heights is still null, so this is a no-op and sizeBuiltGroup() // below is what covers those groups. - applyIntrinsicSize(section, group.entries.length, bands, build.heights); + // + // The rows are counted apart from the bands. A band is an entry as well as a row, so + // a count of entries would price every band twice — once at the row height and once + // at the band's own — and the box would stand taller than what it is holding. + var predicted = applyIntrinsicSize(section, group.entries.length - bands, bands, build.heights); + if (!holds) { + // The height the group had while it was holding its rows, brought up to date + // with the content it holds now; the prediction otherwise. A group that has + // never been on screen has no measured height, and the prediction is the best + // answer there is — the same answer the scrollbar is already giving. + holdLedgerHeight(section, heldLedgerHeight(key, group.entries.length - bands, bands, predicted, build.heights)); + } ledger.appendChild(section); + ledgerContent.set(section, { key: key, turn: group.turn, entries: group.entries, bands: bands }); built += 1; } @@ -6776,13 +8044,42 @@

检视

// sizeBuiltGroup then writes that 0 onto it. A group sized 0px occupies no space, so it // can never come back into the viewport, so it is never re-rendered: the ledger stays // collapsed for good and the document is two thousand pixels tall above a hundred and - // forty kilobytes of rows. It only showed up in the default window, where 200 records + // forty kilobytes of rows. It only showed up in the default window, where PAGE_STEP records // span fewer turns than LEDGER_BATCH_GROUPS and therefore the *whole* ledger is this // batch; the A/B that measured it ran on 显示全部, where the batch is long enough to sit in the // viewport and get measured for real. if (!build.heights) { build.heights = calibrateRowHeights(ledger); - var firstBatch = ledger.querySelectorAll('.turn-group'); + // The visit path prices the difference a released group has with the same + // calibration, so the last one is kept where the visit path can reach it. + ledgerCalibration = build.heights; + // The groups that were built without their rows have nothing to be sized from, and a + // group left unsized collapses to nothing the moment the class goes on. They are + // sized now, from the calibration, with the same arithmetic every other group is + // sized with. + var dormant = ledger.querySelectorAll('.turn-group[data-rows="released"]'); + for (var d = 0; d < dormant.length; d += 1) { + var content = ledgerContent.get(dormant[d]); + if (!content) continue; + // The rows are counted apart from the bands, for the same reason as above: a band + // is an entry as well as a row, and the prediction prices it once, at its own + // height. + var dormantRows = content.entries.length - content.bands; + var dormantPredicted = applyIntrinsicSize(dormant[d], dormantRows, content.bands, build.heights); + // And the height that is held is the measured one, brought up to date with what + // the group holds now — the same rule the build applies to the groups it writes + // above. A group that was never measured has nothing to bring up to date, and the + // prediction is the answer there. + // + // Writing the prediction here instead would undo the rule on every group of the + // first batch, and in the default view the first batch is the whole window: the + // page opens on the newest PAGE_STEP records, and that window spans fewer turns than + // LEDGER_BATCH_GROUPS. + holdLedgerHeight(dormant[d], heldLedgerHeight(content.key, dormantRows, content.bands, dormantPredicted, build.heights)); + } + // The groups that were holding their rows are the ones that can be measured. A group + // without rows measures as its header, and writing that down would collapse it. + var firstBatch = ledger.querySelectorAll('.turn-group[data-rows="live"]'); for (var f = 0; f < firstBatch.length && f < LEDGER_BATCH_GROUPS; f += 1) { sizeBuiltGroup(firstBatch[f]); } @@ -6802,6 +8099,10 @@

检视

// measuring has to stay the last word, and a trailing statement is the one place where // a future synchronous call would quietly measure an empty page. flushDeferredPoll(); + // The groups are watched from here, so a group the reader scrolls to gets its rows back + // before the next paint. Before this the sections do not exist — the table was empty a + // moment ago — and an observer pointed at nothing sees nothing. + watchLedgerGroups(); // Measured here rather than at the end of render(): this is the one exit every path that // changes what the table shows goes through — the filters that call render() and the nine // that call renderLedger() directly. A filter that grows the band therefore cannot leave @@ -6814,10 +8115,10 @@

检视

} // Windows of a few groups never reach the yield: the while loop inside buildStep consumes - // all of them and falls straight through, so the default 200-row table is built exactly as + // all of them and falls straight through, so the default window's table is built exactly as // it always was, in one pass, with no progress written and no frame spent. Seeding the // progress only when there is more than one batch is what keeps that true — otherwise the - // common path paid for a 「正在显示 0 / 200 条」 that no reader ever saw. + // common path paid for a 「正在显示 0 / N 条」 that no reader ever saw. if (groups.length > LEDGER_BATCH_GROUPS) setLedgerProgress(0, buildTotal); buildStep(); } @@ -6918,7 +8219,7 @@

检视

// double-count the shift and leave the page visibly off. // // Guarded, because this button is meaningless in 全部: state.limit is Infinity there, - // and Infinity + 200 is still Infinity, so the click would re-render and change + // and Infinity + PAGE_STEP is still Infinity, so the click would re-render and change // nothing — a control that looks live and is not. It is hidden in that mode, so the // branch is unreachable through the UI; the guard is there so that hiding it can // never be the only thing standing between a reader and a button that does nothing. @@ -6928,7 +8229,7 @@

检视

}); actions.appendChild(more); - // One button for the one choice. It was two — 「只看最新 200 条」 and 「已显示全部」 — + // One button for the one choice. It was two — 「只看最新 N 条」 and 「已显示全部」 — // and the first one relabelled itself to the second one's wording once you were // already showing everything, which made the row read as three controls for two // states. The label names the state you are in; the accessible name names the action. @@ -6981,7 +8282,7 @@

检视

* so follows the panel rather than whatever the last poll happened to build. And it is re-read * on every frame rather than remembered, because 「the end」 moves every poll: the window is * the newest N records and each new one slides it forward, so the bottom of the ledger is a - * different document position every 2.5 seconds. Anything derived from scrollHeight against a + * different document position on every poll. Anything derived from scrollHeight against a * stored value would be correct only between polls. * * Both edges, or neither: an earlier version tested `r.height > 0 || r.top < viewH`, and the @@ -7112,7 +8413,7 @@

检视

* * It moves the page and nothing else. It does not touch `state.limit`, so it cannot make the * window wider — the prohibition this function exists to enforce, and the one the window bar's - * 「已显示 N 条」 depends on. Both ends are already loaded: down reaches the newest 200, up + * 「已显示 N 条」 depends on. Both ends are already loaded: down reaches the newest PAGE_STEP records, up * reaches the oldest of them. */ function setLedgerJumpLabel() { @@ -7260,15 +8561,15 @@

检视

* The mode button names the action; the count beside it names the state. * * They are not saying the same thing twice, and they must not be made to. The count is - * the report — 已显示 200 条, 更早的 9,783 条未加载 — and it is the only thing on the row + * the report — 已显示 N 条, 更早的 M 条未加载 — and it is the only thing on the row * that tells you what you are looking at. The button is the offer, and what it offers is - * by definition not what you already have: from the newest 200, the whole session; from + * by definition not what you already have: from the newest PAGE_STEP records, the whole session; from * everything, back to one page. A button that restates the current state is a button * whose text changes to tell you nothing new. * * It was briefly written the other way and read as a contradiction, because at the time * the label and the count were describing the same quantity with different wording — - * 「只看最新 200 条」 above 「已显示 200 条」. Naming the action dissolves that rather than + * 「只看最新 N 条」 above 「已显示 N 条」. Naming the action dissolves that rather than * causing it: the two now answer different questions, and neither can contradict the * other because neither claims to be the other. * @@ -7286,14 +8587,14 @@

检视

parts.more.textContent = '向上加载更早的 ' + formatInt(Math.min(PAGE_STEP, Math.max(hiddenAbove, 0))) + ' 条'; // The button names the ACTION: it says what pressing it will do, which is the only // information the reader does not already have. The state is already on screen — the - // count beside this button reports 已显示 200 条 or 已显示 9,977 条, and the page title + // count beside this button reports 已显示 N 条 or 已显示 M 条, and the page title // says which turns that covers — so a label restating it spends its width saying - // nothing. At the default the window already holds only the newest 200, so the one + // nothing. At the default the window already holds only the newest PAGE_STEP records, so the one // thing worth offering is the whole session; once everything is shown, the one thing // worth offering is going back to a page. // - // It went the other way twice. Describing the state put 「只看最新 200 条」 next to - // 「已显示 200 条」 — the button echoing the count beside it. Describing the state after + // It went the other way twice. Describing the state put 「只看最新 N 条」 next to + // 「已显示 N 条」 — the button echoing the count beside it. Describing the state after // a second change put 「已显示全部」 on a page showing everything, which is true and // therefore tells you nothing about the button. Naming the action is what both cases // need. @@ -7316,6 +8617,16 @@

检视

/** The topmost row that intersects the viewport, as {id, top}, or null. */ function firstVisibleRow() { + // The group the reader is on has to be holding its rows before the anchor can be read out + // of it. A released group has no rows, and an anchor read from the next live group down + // is an anchor to the wrong place — the reader would be moved to wherever the first group + // that still has rows happens to start. + var sections = dom.ledger.querySelectorAll(':scope > section.turn-group'); + for (var s = 0; s < sections.length; s += 1) { + if (sections[s].getBoundingClientRect().bottom <= 0) continue; + if (sections[s].getAttribute('data-rows') !== 'live') fillLedgerGroup(sections[s]); + break; + } var nodes = dom.ledger.querySelectorAll('[data-row-id]'); for (var i = 0; i < nodes.length; i += 1) { var rect = nodes[i].getBoundingClientRect(); @@ -7403,13 +8714,24 @@

检视

/** * A group's predicted height from its own shape: one head, then rows and bands. * + * The rows and the bands are two counts: `rows` is the number of rows ALONE, and the + * compaction bands are counted in `bands`. A band is an entry as well as a row, so + * passing the entry count as the row count would price every band twice — once at the + * row height and once at the band's own — and the box would stand taller than what it + * is holding. + * * `auto` on the left is what lets the browser prefer the size it has already rendered once * the reader has scrolled past — so scrolling down and back does not accumulate drift. + * + * The height is returned as well as written, because a group that is not holding its rows + * needs the same number on its `min-height`: `contain-intrinsic-size` sizes a group the + * browser has skipped, and a group near the viewport is rendered rather than skipped. */ function applyIntrinsicSize(section, rows, bands, heights) { - if (!heights) return; + if (!heights) return 0; var h = heights.head + rows * heights.row + bands * heights.band; section.style.containIntrinsicSize = 'auto ' + Math.round(h) + 'px'; + return h; } /** @@ -7661,7 +8983,7 @@

检视

// Like every other filter, and for the same reason: the window the reader was looking // at was chosen for a set of records that no longer exists. Setting a range used to - // force 「全部」 instead, on the theory that a 200-cap inside a range silently drops + // force 「全部」 instead, on the theory that the default cap inside a range silently drops // the start of what the reader asked for. Measured, that theory is not worth its // price — a range covering the whole session rendered ~11,000 rows and took the // document to 414,000px, where one query took 19s. The window bar still says how many @@ -7758,7 +9080,7 @@

检视

/* ---------------- inspector ---------------- */ var INSPECTOR_DOCK_QUERY = '(max-width: 1080px)'; - // The height the panel gets when the reader has not asked for another one. 2/5 of the + // The height the panel gets when the reader has not asked for another one. INSPECTOR_DOCK_SHARE of the // window, on the reviewer's instruction (it was a quarter for a while before that). // // One constant, because this number used to be written twice — applyInspectorDock() and @@ -7833,7 +9155,7 @@

检视

* flow — the panel is in the document below the ledger. Where it starts, and where * 「滚到最底部」 brings it back. This one is a position, not a height. * folded — pinned, down to its own header and nothing else. - * docked — pinned to the bottom of the viewport at two fifths. + * docked — pinned to the bottom of the viewport at INSPECTOR_DOCK_SHARE. * full — pinned, uncapped, because the reader asked with the handle. * * Three of those four are pinned and one is not, and the fourth one did not exist for a while: @@ -7893,7 +9215,7 @@

检视

// take is read off the window and the offsets syncStickyOffsets() just measured, and // written here. // - // panelMax is what is left after the pinned strips *and* the ledger's guaranteed quarter. + // panelMax is what is left after the pinned strips *and* the ledger's guaranteed share. // It is the ceiling for all three states, so the expanded panel stops short of covering // the rows and a drag cannot grow past it either. // @@ -7920,9 +9242,9 @@

检视

); var panelMax = Math.max(INSPECTOR_MIN_DOCK_H, inspectorUnderPins() - ledgerReserve); dom.inspectorPanel.style.setProperty('--inspector-full-h', panelMax + 'px'); - // A quarter of the measured window, or the height the reader dragged to — clamped to the - // same ceiling the expanded state gets, so the two ways of making the panel taller cannot - // disagree about where the top one is. + // The dock's share of the measured window, or the height the reader dragged to — clamped + // to the same ceiling the expanded state gets, so the two ways of making the panel taller + // cannot disagree about where the top one is. var dockH = state.inspectorDockH; if (dockH === null || dockH === undefined || !isFinite(dockH)) { dockH = inspectorDefaultDockHeight(); @@ -8425,24 +9747,45 @@

检视

var rawCache = {}; var rawPending = {}; - function rawCacheKey(index) { + function rawCacheKey(index, version) { var payload = state.payload; var session = payload && payload.session ? payload.session : null; var sid = session && typeof session.id === 'string' ? session.id : ''; - var gen = state.generation === null ? 'all' : String(state.generation); + // The slice the value belongs to is the slice the body on screen is, for the same + // reason the session is: a record index is a position inside one body, and the body on + // screen is not always the body the reader last selected. + var gen = displayedGeneration() === null ? 'all' : String(displayedGeneration()); // The validator belongs in this key. A record index is only a position within one version // of one body: the same index in the next version of the session is a different message. // Without the tag, a body that changed since the raw was read kept being served from here // forever, and the page had no way to notice. Pinned by tests/verify-raw-lazy.mjs. - return sid + '#' + gen + '#' + (state.etag || '') + '#' + index; + // + // And the RECORD's own version belongs in it too. This key is what decides whose answer a + // caller gets, and it used to be built out of what is on screen — the validator, the slice, + // the position — and none of those say which RECORD is being read. So a read that is + // retried after a reload found the pending read of the record that had taken its place + // (the rebuild of the panel starts one) and took its answer for its own: the reader + // opened ALPHA and the page handed back CHARLIE. The version is the caller's, not the + // screen's — the same rule the request itself follows. + return sid + '#' + gen + '#' + (state.etag || '') + '#' + index + '#' + (version || ''); } - function requestRaw(index) { + function requestRaw(index, version) { var url = '/api/trajectory/raw?index=' + encodeURIComponent(String(index)) - + '&session=' + sessionParam(); - if (state.generation !== null) { - url += '&generation=' + encodeURIComponent(String(state.generation)); - } + + '&session=' + displayedSessionParam(); + var generation = displayedGeneration(); + if (generation !== null) { + url += '&generation=' + encodeURIComponent(String(generation)); + } + // WHICH RECORD. The index is a position inside the body on screen, and the service may be + // holding a newer body in which that position is another record — so the version of the + // record the reader opened travels with the request, and the service can tell the two + // apart instead of serving the newer record under the older one's number. + // + // The version is the one the RECORD carries, and that is the whole point: a read that is + // retried after a reload must ask about the same record it asked about the first time, or + // the retry is a different question wearing the same question's clothes. + if (version) url += '&v=' + encodeURIComponent(version); return fetch(url, { headers: { Accept: 'application/json' } }).then(function (response) { return response.json().then( function (body) { return { ok: response.ok, body: body }; }, @@ -8470,10 +9813,15 @@

检视

function fetchRaw(record, retried) { var index = record && record.index; if (!isCount(index) || index < 1) return Promise.reject(new Error('record_index_missing')); - var key = rawCacheKey(index); + // The version of the record the READER opened, read off that record and not off the + // screen — and it decides the key, because the key is what says whose answer a caller + // gets. A version read off the screen at this moment would belong to the record that has + // taken this one's place, and the retry would be answered with it. + var version = recordVersionOf(record); + var key = rawCacheKey(index, version); if (Object.prototype.hasOwnProperty.call(rawCache, key)) return Promise.resolve(rawCache[key]); if (rawPending[key]) return rawPending[key]; - var attempt = requestRaw(index).then(function (result) { + var attempt = requestRaw(index, version).then(function (result) { delete rawPending[key]; if (result.ok && result.body && Object.prototype.hasOwnProperty.call(result.body, 'raw')) { var entry = { raw: result.body.raw, omitted: result.body.omitted === true }; @@ -8528,12 +9876,17 @@

检视

var detailCache = {}; var detailPending = {}; - function requestDetail(index) { + function requestDetail(index, version) { var url = '/api/trajectory/record?index=' + encodeURIComponent(String(index)) - + '&session=' + sessionParam(); - if (state.generation !== null) { - url += '&generation=' + encodeURIComponent(String(state.generation)); - } + + '&session=' + displayedSessionParam(); + var generation = displayedGeneration(); + if (generation !== null) { + url += '&generation=' + encodeURIComponent(String(generation)); + } + // The same record, the same way: the version travels with the request so the service can + // refuse when the position it would answer is a different record than the one opened — and + // it is the record's own version, so a retry after a reload asks about the same record. + if (version) url += '&v=' + encodeURIComponent(version); return fetch(url, { headers: { Accept: 'application/json' } }).then(function (response) { return response.json().then( function (body) { return { ok: response.ok, body: body }; }, @@ -8555,10 +9908,13 @@

检视

function fetchDetail(record, retried) { var index = record && record.index; if (!isCount(index) || index < 1) return Promise.reject(new Error('record_index_missing')); - var key = rawCacheKey(index); + // The same version, and for the same reason: the key is the same key, and it decides + // whose answer a caller gets. + var version = recordVersionOf(record); + var key = rawCacheKey(index, version); if (Object.prototype.hasOwnProperty.call(detailCache, key)) return Promise.resolve(detailCache[key]); if (detailPending[key]) return detailPending[key]; - var attempt = requestDetail(index).then(function (result) { + var attempt = requestDetail(index, version).then(function (result) { delete detailPending[key]; if (result.ok && result.body && typeof result.body.index === 'number') { var entry = { @@ -8752,6 +10108,11 @@

检视

updateHandoffButton(); return; } + // The session is captured with the record, because the two reads below are asynchronous + // and the page can be showing another conversation by the time they answer. A label that + // names the session on screen at that moment would put one conversation's name on + // another conversation's message. + var session = state.payload ? state.payload.session : null; var record = entry.record; dom.handoffBtn.disabled = true; inspectorSay('正在读取原文…'); @@ -8772,7 +10133,7 @@

检视

: fetched.raw; var raw = truncateJson(rawSource, HANDOFF_RAW_LIMIT); var text = buildHandoffMarkdown( - state.payload ? state.payload.session : null, + session, entry.turn, record, raw.text, @@ -8868,7 +10229,7 @@

检视

updateRangeHint(); // The refresh button depends on how far the session has run past the snapshot, which // changes on every poll — it has to be re-decided here or it would offer to refresh - // for records that are already in, or stay hidden while 200 records pile up behind it. + // for records that are already in, or stay hidden while a window of records piles up behind it. syncRangeButtons(); // The picker bounds move with the session, which grows during the very first poll. // Setting them once at startup would leave the calendar capped at whatever the @@ -9025,13 +10386,57 @@

检视

} } + /** + * The reader picked a conversation. + * + * A conversation and a run are both sessions, and both go through the one writer that says + * which session is on screen. Picking a conversation leaves the run pick empty, so the page + * shows the conversation's own trajectory. + */ + function onSessionSelectChange() { + var value = dom.sessionSelect.value; + // A different session has a different generation chain; keeping the old pin would + // ask that session for a generation number it may never have had. + state.generation = null; + if (value === 'latest') { + setFollow(true); + return; + } + selectSession(value); + } + + /** + * The reader picked a run, or the conversation itself. + * + * A run is a session with its own transcript, and the conversation's transcript does not + * contain its turns — so this switches the page to that run's own trajectory. The + * conversation pick stays on the run's parent, which is where it already was. + * + * The first option is the conversation itself: picking it returns to the conversation's + * trajectory, and it is where the pick sits until a run is chosen. + */ + function onChildSelectChange() { + var value = dom.childSelect.value; + state.generation = null; + if (!value) { + var owner = ownerSessionId(); + if (!owner) return; + selectSession(owner); + return; + } + selectSession(value); + } + /* ---------------- wiring ---------------- */ function setFollow(enabled) { state.followLatest = enabled; dom.followBtn.setAttribute('aria-pressed', enabled ? 'true' : 'false'); dom.followState.textContent = enabled ? '开' : '关'; - dom.sessionSelect.value = enabled ? 'latest' : state.sessionId || 'latest'; + // The conversation pick names the conversation, not the run: with a run on screen it + // stays on the run's parent. + dom.sessionSelect.value = enabled ? 'latest' : (ownerSessionId() || 'latest'); + renderChildSelect(); state.signature = ''; state.loading = true; fetchSessions(); @@ -9039,6 +10444,29 @@

检视

announce(enabled ? '已开启跟随最新，正在跟踪最近的会话。' : '已关闭跟随最新，固定在所选会话。'); } + /** + * The one place a session is chosen. + * + * A conversation and a sub-agent are both sessions, so both come through here and + * 「which session is on screen」 has one writer and one meaning. A different session has a + * different generation chain, so the pin is dropped here rather than at each call site — a + * generation number the new session may never have had would ask it for a slice it does not + * have. + * + * @param {string} id + */ + function selectSession(id) { + state.generation = null; + state.sessionId = id; + // The groups of the last session are not the groups of this one, and a turn key means + // nothing across two sessions. What is known about them goes with them, so the first + // build of the new session starts from nothing rather than from another session's + // heights. + ledgerState.clear(); + ledgerContent.clear(); + setFollow(false); + } + /** * Builds one button per rule, plus 全部异常 for the set as a whole. Built from the table * rather than written into the markup so that a rule has exactly one name, in one place. @@ -9411,6 +10839,14 @@

检视

window.addEventListener('scroll', scheduleInspectorDock, { passive: true }); window.addEventListener('resize', scheduleInspectorDock); + // The ledger's groups keep the height they were measured at, and a resize changes the + // layout of every group on screen without changing a row or a band in it. The + // measurement is the observer's, and the observer is not told that the window changed + // shape — so the groups the reader is looking at would keep a height of the layout + // before the resize, and the height a released group reserves would be that stale + // number. + window.addEventListener('resize', refreshLedgerMeasurements); + // Two buttons, one job each, and neither writes the field itself — each names the state its // click goes to and hands it to the one writer. The single button this replaced cycled // folded -> half -> expanded -> folded, which was correct and still wrong to use: reaching @@ -9445,18 +10881,8 @@

检视

fetchTrajectory(true); }); - dom.sessionSelect.addEventListener('change', function () { - var value = dom.sessionSelect.value; - // A different session has a different generation chain; keeping the old pin would - // ask that session for a generation number it may never have had. - state.generation = null; - if (value === 'latest') { - setFollow(true); - return; - } - state.sessionId = value; - setFollow(false); - }); + dom.sessionSelect.addEventListener('change', onSessionSelectChange); + dom.childSelect.addEventListener('change', onChildSelectChange); dom.generationSelect.addEventListener('change', function () { var raw = dom.generationSelect.value; diff --git a/plugins/avatasia/mmc-trajectory/miniapp/node/search-segments.mjs b/plugins/avatasia/mmc-trajectory/miniapp/node/search-segments.mjs index ff1ef67c..4d600f9c 100644 --- a/plugins/avatasia/mmc-trajectory/miniapp/node/search-segments.mjs +++ b/plugins/avatasia/mmc-trajectory/miniapp/node/search-segments.mjs @@ -6,7 +6,7 @@ // WHY A SEGMENT AND NOT A FIELD ON THE INDEX. The search surface is five fields per record // (summary, text, result, thinking, toolName) — measured at 36.7 MB for a 44-generation session, // against 8.6 MB for the index. Folding it in would make every read of the index pay for search, on -// a page that reads the index every eight seconds and searches when someone types. Kept apart, the +// a page that reads the index on every poll and searches when someone types. Kept apart, the // ordinary read never opens the surface and a search never opens the index's own metadata twice. // // WHY PER GENERATION AND NOT PER SESSION. A sealed generation's records cannot change, so its @@ -43,8 +43,19 @@ import { createHash } from 'node:crypto'; import { mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises'; import { join } from 'node:path'; -/** Bumped when the on-disk shape changes in a way an older reader cannot survive. */ -export const SEGMENT_SCHEMA_VERSION = 3; +/** + * Bumped when the on-disk shape changes in a way an older reader cannot survive. + * + * 4: every line carries the record's version (`v`). A hit is a POSITION, and a position is only the + * same record in another body if the version says so — so a segment written before this would hand + * the page hits it cannot check, and the page would have to take them on faith. + * + * 5: every manifest states how many of the body's records the segments do NOT cover. A record with + * no locator names no file, so it belongs to no generation's segment — and a set that covers fewer + * records than the body stands for a DIFFERENT set, whose fingerprint the page is not holding. A + * manifest written before this does not say, and a set that does not say cannot be used. + */ +export const SEGMENT_SCHEMA_VERSION = 5; /** * The fields a search reads, in the page's order. @@ -55,9 +66,9 @@ export const SEGMENT_SCHEMA_VERSION = 3; */ export const SEARCH_FIELDS = ['summary', 'text', 'result', 'thinking', 'toolName']; -/** @typedef {{ fileName: string, generation: number | null, size: number, revision: string | null, truncated?: boolean }} SegmentSource */ +/** @typedef {{ fileName: string, generation: number | null, size: number, mtimeMs?: number | null, revision: string | null, truncated?: boolean }} SegmentSource */ -/** @typedef {{ schemaVersion: number, generation: number | null, view: string, recordCount: number, surfaceBytes: number, surfaceDigest: string, source: SegmentSource[] }} SegmentManifest */ +/** @typedef {{ schemaVersion: number, generation: number | null, view: string, recordCount: number, surfaceBytes: number, surfaceDigest: string, uncovered: number, source: SegmentSource[] }} SegmentManifest */ /** * One record's searchable text. @@ -169,6 +180,37 @@ export function fileIdxRowsFromPayload(payload, raws, files) { return out; } +/** + * How many of the body's records no segment covers. + * + * A record is filed under the generation its locator names, and a record with no locator names no + * file — so it is filed under no generation, and no segment holds it. The segments then cover fewer + * records than the body, and a shorter set is a DIFFERENT set: its fingerprint is the fingerprint + * of the records it does cover, and the page is holding the ones it does not. So the count is what + * the manifest states, and a reader that sees it knows the set cannot answer for the body. + * + * A row that is not a placement at all counts as uncovered. The count is read as 「how much is + * missing」, so the safe direction is the one that says something IS missing. + * + * @param {Array<{ fileIdx: number }>} rows one entry per record, from + * {@link fileIdxRowsFromPayload} + * @returns {number} + */ +export function uncoveredRecordCount(rows) { + const list = Array.isArray(rows) ? rows : []; + let uncovered = 0; + for (const row of list) { + if (!row || typeof row !== 'object') { + uncovered += 1; + continue; + } + if (typeof row.fileIdx !== 'number' || !Number.isFinite(row.fileIdx) || row.fileIdx < 0) { + uncovered += 1; + } + } + return uncovered; +} + /** * Builds one generation's searchable lines. * @@ -206,6 +248,10 @@ export function buildSegment(input) { kind: typeof record.kind === 'string' ? record.kind : '', generation: typeof record.generation === 'number' ? record.generation : null, summary: typeof record.summary === 'string' ? record.summary : '', + // The version of the record this line is. It travels with the hit because a hit is a position + // and the page has to be able to tell whether that position still names the same record in the + // body it is showing. + v: typeof record.v === 'string' ? record.v : '', fileIdx: typeof record.fileIdx === 'number' ? record.fileIdx : -1, }) + '\t' + searchSurfaceOf(record)); } @@ -256,11 +302,11 @@ export function scanSegment(surface, needle) { * * @param {string} surface the whole segment text * @param {number[]} ordinals as returned by {@link scanSegment} - * @returns {Array<{ index: number, turnIndex: number | null, kind: string, summary: string, generation: number | null } | null>} + * @returns {Array<{ index: number, turnIndex: number | null, kind: string, summary: string, generation: number | null, v: string } | null>} */ export function readSegmentHits(surface, ordinals) { const wanted = new Set(ordinals); - /** @type {Map} */ + /** @type {Map} */ const found = new Map(); if (wanted.size === 0) return []; let start = 0; @@ -286,6 +332,7 @@ export function readSegmentHits(surface, ordinals) { kind: typeof parsed.kind === 'string' ? parsed.kind : '', summary: typeof parsed.summary === 'string' ? parsed.summary : '', generation: typeof parsed.generation === 'number' ? parsed.generation : null, + v: typeof parsed.v === 'string' ? parsed.v : '', }); } } @@ -376,7 +423,7 @@ export function createSegmentStore(root) { * @param {string} view * @param {string} generationKey * @param {{ surface: string }} segment - * @param {{ generation: number | null, source: SegmentManifest['source'] }} meta + * @param {{ generation: number | null, source: SegmentManifest['source'], uncovered: number }} meta */ async write(sessionKey, view, generationKey, segment, meta) { const dir = dirOf(sessionKey, view, generationKey); @@ -391,6 +438,11 @@ export function createSegmentStore(root) { recordCount: segment.surface === '' ? 0 : segment.surface.split('\n').length, surfaceBytes: Buffer.byteLength(segment.surface), surfaceDigest: digestOf(segment.surface), + // How many of the body's records these segments do NOT cover. Written here so that a reader + // can tell a set that stands for the whole body from one that stands for part of it — see + // `uncoveredRecordCount` for what the count is, and the schema version above for what + // happens to a manifest that does not say. + uncovered: meta.uncovered, source: meta.source, }; await writeFile(`${join(dir, 'manifest.json')}.tmp`, `${JSON.stringify(manifest)}\n`, 'utf8'); @@ -420,6 +472,11 @@ export function createSegmentStore(root) { if (!manifest || typeof manifest !== 'object') return null; if (manifest.schemaVersion !== SEGMENT_SCHEMA_VERSION) return null; if (manifest.view !== view) return null; + // The manifest has to say how many of the body's records these segments do not cover. A + // manifest that does not say cannot be told apart from one that says 「none」, and a set that + // covers part of the body is not the set the page is holding — so a manifest that does not + // say is a manifest this reader cannot use. + if (!Number.isInteger(manifest.uncovered) || manifest.uncovered < 0) return null; if (!sameSource(manifest.source, source)) return null; let surface; try { @@ -440,11 +497,156 @@ export function createSegmentStore(root) { }; } +/** + * The version tokens of a segment's records, in order, up to a limit. + * + * Every line of a segment carries the version of the record it is, so the versions are already + * there — this reads them out rather than keeping a second copy of them. The reading is by hand + * instead of by parsing each line's whole answer, because a search over a big segment would then + * parse every line it does not need; `tests/verify-search-segments.mjs` pins that this and the + * parsed answer agree. + * + * The route does not read the versions this way: it needs each record's LEDGER position as well, + * so it reads the two in one walk — see {@link segmentLedgerPositions}. + * + * @param {string} surface + * @param {number} [limit] stop after this many records; zero is zero records, and a missing, + * negative or fractional limit is every record + * @returns {Array} + */ +export function segmentVersions(surface, limit) { + /** @type {Array} */ + const out = []; + if (typeof surface !== 'string' || surface === '') return out; + // The same rule as the payload's own fold over the first N records: an explicit limit of zero + // is zero records, and a missing, negative or fractional limit is every record. The two are two + // readings of one fact — the versions of the first N records — so they have to agree. + const wanted = Number.isInteger(limit) && limit >= 0 ? limit : Infinity; + let start = 0; + while (start <= surface.length && out.length < wanted) { + const nl = surface.indexOf('\n', start); + const end = nl === -1 ? surface.length : nl; + const tab = surface.indexOf('\t', start); + if (tab >= 0 && tab <= end) out.push(versionInAnswer(surface.slice(start, tab))); + if (nl === -1) break; + start = nl + 1; + } + return out; +} + +/** + * The ledger position of every record a segment holds, and the version of each. + * + * A line carries the record's TURN and the record's own number in the body, and those two are + * what the ledger's order is made of: the ledger draws turn by turn, and inside a turn the + * records in the order they were emitted. So the two numbers on a line are enough to put that + * record back where the body has it. + * + * Why that matters: the segments are one file per generation and the body is not in file order. + * A record that belongs to an EARLIER turn and arrived after that turn was written lives in a + * LATER file, so the first N records of the body and the first N records of the files are two + * different sets. Reading the versions out of the files alone therefore answers about the wrong + * set, and the answer is refused for a body that did not change. + * + * The version comes along because the same walk has to hand it over: a second walk over the + * same surface would cost the same again, and the answer that needs them is the answer about + * THIS set of records. + * + * The numbers are read by hand, like the version: a search over a big segment would then parse + * every line it does not need. `index` and `turnIndex` are written before the summary, so the + * first one in a line's answer is the field and not something the summary says. + * + * @param {string} surface + * @returns {Array<{ turnIndex: number | null, index: number | null, v: string }>} one entry per + * record, in the order the lines sit in this surface + */ +export function segmentLedgerPositions(surface) { + /** @type {Array<{ turnIndex: number | null, index: number | null, v: string }>} */ + const out = []; + if (typeof surface !== 'string' || surface === '') return out; + let start = 0; + while (start <= surface.length) { + const nl = surface.indexOf('\n', start); + const end = nl === -1 ? surface.length : nl; + const tab = surface.indexOf('\t', start); + if (tab >= 0 && tab <= end) { + const answer = surface.slice(start, tab); + out.push({ + turnIndex: numberInAnswer(answer, '"turnIndex":'), + index: numberInAnswer(answer, '"index":'), + v: versionInAnswer(answer), + }); + } + if (nl === -1) break; + start = nl + 1; + } + return out; +} + +/** + * One number out of a line's answer. + * + * Read by hand for the same reason the version is: parsing the whole answer of every line of a + * big segment is the cost the hand-reading exists to avoid. The number ends at the first + * separator after it, which is where the field ends. + * + * An empty field is nothing, not a number: `Number('')` is zero, and a record that carries no + * number of its own must not come back as the record numbered zero. + * + * @param {string} answer the JSON prefix of one segment line + * @param {string} key the field to read, with its quotes and its colon + * @returns {number | null} the number, or nothing when the field is not there + */ +function numberInAnswer(answer, key) { + const at = answer.indexOf(key); + if (at < 0) return null; + let end = at + key.length; + let value = ''; + while (end < answer.length) { + const ch = answer.charAt(end); + if (ch === ',' || ch === '}') break; + value += ch; + end += 1; + } + const parsed = value.trim() === '' ? NaN : Number(value); + return Number.isFinite(parsed) ? parsed : null; +} + +/** + * The version token inside one line's answer. + * + * The answer is JSON and the version is one of its fields. Reading that one field by name is + * what keeps a search from parsing every line of a segment; `buildSegment` writes the field, and + * the guard pins that the two readings agree. + * + * @param {string} answer the JSON prefix of one segment line + * @returns {string} + */ +function versionInAnswer(answer) { + const at = answer.indexOf('"v":"'); + if (at < 0) return ''; + const from = at + 5; + let end = from; + while (end < answer.length) { + const ch = answer.charAt(end); + // A backslash escapes the next character, so it cannot be the end of the value. + if (ch === '\\') { end += 2; continue; } + if (ch === '"') break; + end += 1; + } + return answer.slice(from, end); +} + /** * Whether the generation a segment was built from is still the generation on disk. * - * Identity, not mtime: a sealed file is not written to, and a reader that treated a touch as a - * change would rebuild correct bytes on every poll. + * Identity is the file name, the generation, the size, the mtime, whether the read was cut short, + * and the revision when both sides carry one: all of them have to agree for the segment to still + * be about the same bytes. + * + * The mtime IS part of that, deliberately. The parsed-rows cache and the trajectory entry both + * key on (size, mtime), so a file rewritten in place — the same length, different bytes — is a + * different file to everything else in here, and a touch is a change here as much as it is there. * * @param {SegmentManifest['source']} stored * @param {SegmentManifest['source']} live @@ -459,6 +661,15 @@ export function sameSource(stored, live) { if (a.fileName !== b.fileName) return false; if (a.generation !== b.generation) return false; if (a.size !== b.size) return false; + // The mtime is this process's idea of a file's identity: the parsed-rows cache and the + // trajectory entry both key on (size, mtime), so a file that was rewritten IN PLACE — the same + // length, different bytes — is a different file to everything else in here. Leaving the mtime + // out of this comparison is what let a same-size rewrite keep a segment built from the old + // bytes, and the search then answered from those bytes as if they were the current ones. + // + // A sealed file is not written to, so its mtime does not move on its own; comparing it costs + // a rebuild only when something touched a file it had no reason to touch. + if (a.mtimeMs !== b.mtimeMs) return false; // Same reason as in the index: a generation read short by our own cap produces a segment that // covers a prefix, while `size` is the whole file. Trusting it would answer with a smaller // `total` and say nothing about being wrong. diff --git a/plugins/avatasia/mmc-trajectory/miniapp/node/server.mjs b/plugins/avatasia/mmc-trajectory/miniapp/node/server.mjs index 37b872c5..0c2e865b 100644 --- a/plugins/avatasia/mmc-trajectory/miniapp/node/server.mjs +++ b/plugins/avatasia/mmc-trajectory/miniapp/node/server.mjs @@ -3,7 +3,7 @@ import { open, readdir, readFile, stat } from 'node:fs/promises'; import { existsSync } from 'node:fs'; import { createServer } from 'node:http'; -import { basename, join, resolve, sep } from 'node:path'; +import { basename, isAbsolute, join, relative, resolve, sep } from 'node:path'; import { homedir } from 'node:os'; import { createHash } from 'node:crypto'; import { @@ -18,7 +18,9 @@ import { normaliseWhitespace, readSegmentHits, scanSegment, + segmentLedgerPositions, searchSurfaceOf, + uncoveredRecordCount, } from './search-segments.mjs'; /** @typedef {import('./miniapp-api.js').MiniAppContext} MiniAppContext */ @@ -54,8 +56,8 @@ const GENERATION_PROBE_BYTES = 64 * 1024; * * The last clause used to end "for counts nobody treats as exact", and that is what turned out * to be false: the page printed the message count in the session picker (`（3 分钟前 · 10 条）`), so it - * was treated as exact. On a session whose active file is 1 MB, the first 64 KB held 10 messages - * out of 23,642 — a number that was wrong by three orders of magnitude and carried no hint of it. + * was treated as exact. When the prefix was a small part of the file, the count it produced was + * wrong by orders of magnitude and carried no hint of it. * So the prefix stays bounded and the counters it buys are only REPORTED when the prefix was the * whole file; see {@link describeSession}. Counting exactly is not available: the list reads the * active file, and an exact count there would still miss the rest of the lineage. @@ -72,14 +74,40 @@ const MAX_RAW_BYTES = 65536; * This has to stay at or above the page's poll interval, and that relationship — not this * number — is the invariant tests/verify-list-cache.mjs holds. Below the interval the cache * cannot survive a single poll: every tick expires it, so the list re-walks the session tree - * and reopens the runtime database on every tick. Measured on this machine, that was most of - * the 16 GB this process read in 68 minutes, and the allocation behind it held the process at - * a 1.2-2.5 GB high-water mark for as long as the page stayed open. + * on every tick. The walk is the expensive part of serving the list, and a cache that expires + * on every tick pays for it on every tick, for as long as the page stays open. + * + * The runtime database is a separate matter and this TTL does not govern it. The titles read + * rides the same interval, but the current conversation is re-read on every poll and is not + * cached at all: it is the one fact here that must not be stale, because a cached "current + * conversation" is the very defect the picker was built to remove. That read costs a few + * milliseconds — measured at 4.00 ms per call against the poll interval, a fraction of one percent of it — + * which is why it is left uncached rather than paid for with a TTL that could hide a switch + * from one conversation to another. */ const LIST_CACHE_TTL_MS = 3000; const SESSION_CACHE_TTL_MS = 5000; const SESSION_CACHE_ENTRIES = 80; const TRAJECTORY_CACHE_ENTRIES = 4; +/** + * How many bytes one piece of the trajectory body may hold. + * + * The body of a large session is tens of megabytes, and it used to be built by ONE + * `JSON.stringify` call — so the process had to hold all of it as one string at once, and V8's + * string builder leaves its intermediate copies behind while it doubles its buffer. + * + * The numbers here were measured on 2026-10-23 on this machine, on the code as it stood before + * the body was chunked, on one session of 57,436 records whose body was 53,581,503 bytes + * (52,648,733 characters): serializing it cost 749 MB of heap on top of the 366 MB payload it + * came from, and with every cache emptied the process still held 1,070 MB of resident memory on + * an 8.4 MB live heap. + * + * A megabyte is a compromise. Small enough that the builder's doubling is noise next to the + * payload it is serializing; large enough that a 53 MB body is fifty-odd writes rather than + * fifty thousand. A body piece that cannot be made smaller than this is one element bigger than + * the budget, and the budget does not pretend otherwise: it bounds that element instead. + */ +const TRAJECTORY_BODY_CHUNK_BYTES = 1024 * 1024; /** * The most hits one search response carries. The count is still reported in full — a needle that * matches ten thousand records is a real answer — so this is a transport bound and not a claim @@ -87,39 +115,110 @@ const TRAJECTORY_CACHE_ENTRIES = 4; */ const MAX_SEARCH_HITS = MAX_HITS; /** - * Upper bound on the parsed generation rows kept resident, in bytes. + * How many BYTES OF ACCOUNTING the parsed generation rows may cost. + * + * The unit is NOT resident bytes, and this comment used to say it was. V8 cannot report an object + * graph's size, so an entry is charged for the SOURCE bytes it was parsed from, times + * `PARSED_ROWS_HEAP_PER_SOURCE_BYTE`. What this budget bounds is therefore how much INPUT the + * cache holds, not how much memory the process ends up holding. The two are related by a measured + * factor — the rows cost more than the bytes they were parsed from — and that factor moves with + * the shape of the messages, so it is a measurement and not an identity. * * This budget no longer covers the file that matters. A cache keyed on path+size+mtime for a file * that is being appended to has no hit rate to speak of: `handleTrajectory` only reaches the parse * after the trajectory key — which carries the same size and mtime — has already missed, so every * store here is followed by a miss and the entry is dead weight that still costs its rows' - * memory. Measured on this machine, retaining it was the largest single cost in the process: 378 MB - * of heap after one session and 304 polls, against 145 MB once it is not retained. + * memory. * * What is left is the sealed generations, whose stat never changes and which are therefore the - * only parses a second request can genuinely reuse. Those files are rare at a session root — the - * store keeps rotated generations under `snapshots/`, not beside `messages.jsonl` — so the budget - * is now generous rather than tuned. Pinned by tests/verify-parsed-rows.mjs. + * only parses a second request can genuinely reuse. A LONG session has a great many of them — the + * store keeps one rotated generation per compaction under `snapshots/` — and when they add up to + * more than this budget, the ones the budget cannot hold are parsed again on every request. So + * this is a trade and not a generosity: what the budget holds is paid for in memory, and what it + * does not hold is paid for again on every poll. + * + * `PARSED_ROWS_CACHE_TTL_MS` bounds the other end: how long the retained rows stay after nobody + * asks for them. Pinned by tests/verify-parsed-rows.mjs. */ const PARSED_ROWS_CACHE_BYTES = 256 * 1024 * 1024; /** * Heap charged to one cache entry per source byte read. V8 cannot report an object graph's * size, so a byte budget needs a proxy. * - * Measured on this machine's own lineage: 95.3 MB of messages.jsonl retained 116 MB of heap - * after two forced collections, i.e. 1.22x the source. 2x leaves headroom over that while - * still erring on the side of over-counting, because a proxy that under-counts would let the - * cache grow past its own budget without noticing. + * The rows a session leaves in the cache cost more than the bytes they were parsed from — the + * multiple has been measured between 1.2x and 1.5x — and it moves with the shape of the messages, + * so PARSED_ROWS_HEAP_PER_SOURCE_BYTE leaves headroom over it while still erring on the side of + * over-counting: a proxy that under-counts would let the cache grow past its own budget without + * noticing. * * It was 3x before, and that was wrong in the direction that matters: 95.3 MB x 3 charged - * 286 MB against a 256 MB budget, so the LRU evicted entries it had room for in reality and + * 286 MB against the budget, so the LRU evicted entries it had room for in reality and * re-parsed them on the next poll. The budget is only as good as this number. */ const PARSED_ROWS_HEAP_PER_SOURCE_BYTE = 2; +/** + * How long the parsed rows of a generation stay in the cache after the last request that wanted + * them. + * + * The rows are the largest thing this service holds, and they are only worth holding while + * somebody is asking for them. The page rebuilds the whole trajectory on every poll while it is + * open, and it stops asking the moment the tab is hidden or closed — so a cache with no expiry + * keeps a conversation's rows for as long as the runtime lives, which is how the last session a + * reader opened stayed resident after they walked away. + * + * The shipped limit has two numbers behind it. The page polls as often as the client declares + * (`TRAJECTORY_POLL_INTERVAL`), so an open tab asks again long before this expires. + * And the page stops polling the moment its tab is HIDDEN, so a reader who switches away for a + * while is the ordinary case and not an exception — a limit shorter than that would make every + * return to the tab cost a re-parse. A reader who is really gone gets their rows back. + * + * **WHEN THEY COME BACK IS NOT THE SAME MOMENT AS WHEN THEY EXPIRE.** The limit is when the rows + * stop being wanted; the release happens on the next sweep, so the rows are gone no later than + * one sweep interval after the limit, with the shipped limit and the shipped sweep interval. + * The limit and the interval have to move together: the interval is at most a quarter of the + * limit, so the release is no later than a quarter of the limit after it; and when the limit is + * so small that a quarter of it is less than a millisecond, the interval is not finer than a + * millisecond and the release is at most a millisecond late. Both of those are asserted. + * + * An expiry is a MISS and not a deletion of something still wanted: the next request parses the + * file again and the cache fills itself back up. The price of that is one re-parse, and only when + * the whole-session cache misses too — which happens only when the conversation grew while the + * reader was away, in which case the rebuild was going to happen anyway. + * + * The timer that does the releasing is unref'd, so it never keeps the process alive, and it is + * stopped when the cache empties. Pinned by tests/verify-parsed-rows.mjs. + */ +const PARSED_ROWS_CACHE_TTL_MS = 10 * 60 * 1000; +/** + * How often the cache sweeps for what has expired. + * + * The release lands on a tick, so this interval is also the gap between "the rows expired" and + * "the rows are gone" — which is why it is a named constant and not a fraction written at the + * call site: the promise to a reader is the limit PLUS this, and the two have to stay in that + * relation. + * + * The relation is enforced where the interval is TAKEN, not only here: the interval is clamped + * to a quarter of the limit the cache was given, so "never later than a quarter of the limit + * beyond it" holds for every limit above four milliseconds and not only for the shipped one. A + * constant that could outgrow the limit would make the promise true on the day it was written + * and false after somebody changed the limit. + * + * The timer that runs the sweep is started with this interval, and the cache reports the interval + * it uses, so the promise above is checked against the running cache rather than against two + * numbers in the source. The call that starts the timer is read by a check of its own, so the + * promise cannot drift away from the timer that has to keep it. + * + * The clamp has a floor of one millisecond, because a timer cannot be finer than that. For a + * limit below four milliseconds the floor is the larger of the two, and the release is then at + * most a millisecond late rather than a quarter of the limit late. + * + * Pinned by tests/verify-parsed-rows.mjs. + */ +const PARSED_ROWS_CACHE_SWEEP_INTERVAL_MS = 150 * 1000; /** * How many decoded messages one trajectory entry keeps for `/raw`, per open session. * - * Bounded by count rather than bytes because the thing being kept is one message, and a 64 KB cap + * Bounded by count rather than bytes because the thing being kept is one message, and the raw cap * is already what decides whether a message is too big to show. The point is not to be a cache — * it is so that paging back and forth in the inspector does not re-read the file each time — and a * reader who opens more than this many at once is served correctly either way. @@ -131,7 +230,7 @@ const JSON_HEADERS = { 'content-type': 'application/json; charset=utf-8', 'cache * Scans for the list counters, one per field. Each one stops right after the opening quote of * the value, so the value can be read out of `text` by position instead of being captured — * which is the whole point: a capturing `matchAll` allocates a match object per hit, and on a - * 64 KB prefix with thousands of messages that is thousands of short-lived objects per session, + * the peek prefix with thousands of messages that is thousands of short-lived objects per session, * per rebuild. * * They carry `/g` and are driven with `test`, which allocates nothing. That mutates `lastIndex`, @@ -338,6 +437,276 @@ export function lineDigest(bytes) { return createHash('sha256').update(bytes).digest('hex').slice(0, LINE_DIGEST_CHARS); } +/** + * The version of ONE record: what makes index N in one body the same record as index N in another. + * + * A record index is a POSITION, and a position means nothing without the body it was counted in. + * Two bodies of the same session at the same slice can hold different records at the same index — + * a transcript rewritten under a running page is the ordinary way that happens — and a reader who + * is shown 「命中 1 条」 pointing at a row whose text does not contain the needle has been told + * something false. + * + * So a version belongs to a position read the same way the session and the slice do: a request + * that names an index names the record it means, and the service can tell the two outcomes apart + * instead of serving the newer body under the older body's number. + * + * It is built from the LINE's digest and the record's position inside that line, and from nothing + * else. That matters twice over: + * + * - The line digest is already computed for every locator, so this costs no extra reading. + * - It is stable under APPEND. A session that grows does not rewrite the lines it already has, so + * the records a reader is looking at keep their version. Otherwise every poll would expire every + * row on screen, and the page would spend its life refusing its own answers — the common case + * would be the broken one. + * + * @param {string | null | undefined} digest the line's digest + * @param {number | null | undefined} rowPos the record's position inside that line + * @returns {string} + */ +export function recordVersionToken(digest, rowPos) { + const line = typeof digest === 'string' ? digest : ''; + const pos = Number.isInteger(rowPos) ? rowPos : 0; + return createHash('sha256').update(`${line}:${pos}`).digest('hex').slice(0, 16); +} + +/** + * Stamps every record of a built payload with its version. + * + * One place, because there are two callers that must agree: the body the page reads, and the + * search answer that filters it. A body whose records carry one version and an answer whose hits + * carry another is two answers to 「这条记录是哪一版」, and the page is then the only place that + * could notice — which is exactly where the check must not live. + * + * @param {{ turns?: Array }} payload + * @param {Map} locators record index -> its line's locator + * @returns {typeof payload} + */ +export function stampRecordVersions(payload, locators) { + const turns = payload && Array.isArray(payload.turns) ? payload.turns : []; + for (const turn of turns) { + const records = turn && Array.isArray(turn.records) ? turn.records : []; + for (const record of records) { + if (!record || typeof record !== 'object') continue; + const locator = locators && typeof locators.get === 'function' ? locators.get(record.index) : undefined; + record.v = recordVersionToken(locator ? locator.digest : null, record.rowPos); + } + } + return payload; +} + +/** + * The identity of a SET of records: a fold over their version tokens, in the order they are shown. + * + * A version token says 「this one record is this one version」. It does not say anything about a + * set of them, and a set is what a search answer is about: an answer says 「these records match and + * the rest do not」, and the part that matters to the reader is the part it does NOT name. + * + * Two properties, and both of them are the point: + * + * - **It is stable under APPEND.** A session that grows does not change the records it already + * has, so the fold over the first N records is the same before and after an append. This is + * the same reason the version token is built from the line's digest rather than from the body: + * a body-level identity (an ETag, a digest of the whole body) changes every time the session + * is written to, and then every answer stops applying every few seconds — the common case + * would be the broken one. + * - **It changes when a record is rewritten.** The token is a digest of that record's own bytes, + * so a record whose text was replaced under the page has a different token, and the fold over + * the same positions has a different answer. + * + * The fold is ordered, so a set that was reordered is a different set. Two lanes of 32 bits each: + * a collision is not a thing a reader can notice, and one lane of 32 is one thing a reader could + * notice. + * + * @param {Array} versions the records' version tokens, in order + * @returns {string} 16 hex characters + */ +export function recordSetFingerprint(versions) { + let h1 = 0x811c9dc5 >>> 0; + let h2 = 0x9e3779b9 >>> 0; + const list = Array.isArray(versions) ? versions : []; + for (let i = 0; i < list.length; i += 1) { + const text = typeof list[i] === 'string' ? list[i] : ''; + for (let j = 0; j < text.length; j += 1) { + const code = text.charCodeAt(j); + h1 = Math.imul(h1 ^ code, 16777619) >>> 0; + h2 = Math.imul(h2 ^ code, 2246822519) >>> 0; + } + // A separator, so that ["ab", "c"] and ["a", "bc"] are two different sets. + h1 = Math.imul(h1 ^ 0x1f, 16777619) >>> 0; + h2 = Math.imul(h2 ^ 0x1f, 2246822519) >>> 0; + } + return hex8(h1) + hex8(h2); +} + +/** + * Eight hex characters, most significant first. + * @param {number} value + * @returns {string} + */ +function hex8(value) { + return (value >>> 0).toString(16).padStart(8, '0'); +} + +/** + * The version tokens of a payload's records, in the order the ledger shows them. + * + * The order is the payload's, not the index order: the ledger draws turn by turn, and a fold over + * a different order is a different answer to 「这批记录是哪一批」. + * + * @param {{ turns?: Array }} payload + * @param {number} [limit] stop after this many records; zero is zero records, and a missing, + * negative or fractional limit is every record + * @returns {Array} + */ +export function payloadRecordVersions(payload, limit) { + /** @type {Array} */ + const out = []; + const cap = Number.isInteger(limit) && limit >= 0 ? limit : Infinity; + const turns = payload && Array.isArray(payload.turns) ? payload.turns : []; + for (const turn of turns) { + const records = turn && Array.isArray(turn.records) ? turn.records : []; + for (const record of records) { + if (!record || typeof record !== 'object') continue; + // The cap is checked BEFORE the push, so a limit of zero is zero records. Checked after the + // push it would yield one version, and the fold over the page's empty set would then never + // match the service's own fold over its empty set — the request would be refused for ever. + if (out.length >= cap) return out; + out.push(typeof record.v === 'string' ? record.v : ''); + } + } + return out; +} + +/** + * The ledger's order: the turns in the order the body numbers them, and inside a turn the + * records in the order they were emitted. + * + * ONE definition, because two answers have to agree on it. The fold over the first N records + * says which SET of records an answer is about, and the order the hits are reported in says how + * that answer reads. Both are the body's order — the ledger draws turn by turn — so both are + * this comparison. + * + * The record's own number is not the order. It is the order the records were EMITTED in, and a + * record that belongs to an earlier turn and arrived after that turn was written is emitted + * later than a record of a later turn: its number is larger, and it still comes first. That is + * why the turn comes first here, and why sorting by the number alone is the wrong order. + * + * @param {{ turnIndex?: number | null, index?: number | null }} a + * @param {{ turnIndex?: number | null, index?: number | null }} b + * @returns {number} + */ +export function byLedgerOrder(a, b) { + // A record with no turn of its own cannot be placed, and the callers that read these fields + // have already refused such a record. The fallback is here so that a comparison can never + // read `null` as a number. + const turnA = typeof a.turnIndex === 'number' ? a.turnIndex : Number.MAX_SAFE_INTEGER; + const turnB = typeof b.turnIndex === 'number' ? b.turnIndex : Number.MAX_SAFE_INTEGER; + if (turnA !== turnB) return turnA - turnB; + const indexA = typeof a.index === 'number' ? a.index : 0; + const indexB = typeof b.index === 'number' ? b.index : 0; + return indexA - indexB; +} + +/** + * The version tokens of the first N records, in the order the ledger shows them, read from the + * stored segments rather than from a rebuilt body. + * + * THE SAME FACT AS {@link payloadRecordVersions}, read the other way round. The body is in + * ledger order — turn by turn, and inside a turn the records in the order they were emitted — + * and the segments are not: they are one file per generation, read oldest first. A record that + * belongs to an EARLIER turn and arrived after that turn was written lives in a LATER file, so + * the first N records of the body and the first N records of the files are two different sets. + * + * That is not a detail of the fixture: a turn that spans a compaction is written like that, and + * a fold over the files answers about a set the page is not holding. The fingerprint then does + * not match the one the page asked for, and the service refuses a body that did not change. + * + * The positions on the lines are what puts the records back: the turn a record is in, and the + * record's own number in the body. Both are on the line, so nothing has to be rebuilt to know + * them — and the order is the body's, because the body numbers its turns in the order it draws + * them and its records in the order it emits them. + * + * `null` is the answer for a record that cannot be placed: a line that carries no turn or no + * record number cannot be put back into the ledger's order, and an answer about a set we cannot + * name is not an answer about the set the page holds. The caller falls back to the rebuild, + * which is the same answer it would have given before. + * + * @param {Array<{ turnIndex: number | null, index: number | null, v: string }>} positions the + * ledger position of every record the segments cover + * @param {number} [limit] stop after this many records; zero is zero records, and a missing, + * negative or fractional limit is every record + * @returns {Array | null} + */ +export function segmentRecordVersions(positions, limit) { + const list = Array.isArray(positions) ? positions : []; + /** @type {Array<{ turnIndex: number, index: number, v: string }>} */ + const placed = []; + for (const one of list) { + if (!one || typeof one !== 'object') return null; + const turnIndex = one.turnIndex; + const index = one.index; + if (typeof turnIndex !== 'number' || !Number.isFinite(turnIndex)) return null; + // A negative number is how the segment says 「this record has no number of its own」, the + // same way it says a record belongs to no file. Such a record cannot be placed inside its + // turn, and a fold that put it somewhere would be folding over a set nobody can name. + if (typeof index !== 'number' || !Number.isFinite(index) || index < 0) return null; + placed.push({ turnIndex, index, v: typeof one.v === 'string' ? one.v : '' }); + } + // The ledger's order: the turns in the order the body numbers them, and inside a turn the + // records in the order they were emitted. The record's own number is what says which is + // which inside a turn — the segments hold one turn's records in more than one file. + placed.sort(byLedgerOrder); + // The same rule as the payload's own fold over the first N records: an explicit limit of zero + // is zero records, and a missing, negative or fractional limit is every record. The two are two + // readings of one fact — the versions of the first N records — so they have to agree. + const wanted = Number.isInteger(limit) && limit >= 0 ? limit : Infinity; + return placed.slice(0, wanted).map((one) => one.v); +} + +/** + * Whether a tail may be answered to a page that holds a particular set of records. + * + * A tail carries the records AFTER the page's cursor. The records before it are not in the tail + * at all — the page keeps the copies it has — so the tail is only about the page's copy if that + * copy is still the service's. + * + * A record rewritten in place keeps its index and its id, so the cursor still points at the same + * record, the tail looks empty, and the page merges nothing: it keeps the old text and installs + * the new body's validator. Nothing else in the protocol can see it, which is why the tail names + * the set it is about. + * + * So the page says which set it holds — how many records, and a fingerprint of their versions — + * and the service folds its own first N the same way. The two have to agree before a tail is + * answered with a tail. The count is the FIRST N for the reason the search has it: a page that is + * behind because the session grew has fewer records, not different ones. + * + * `null` means the tail may be answered. Otherwise the refusal, with the status and the code the + * page answers it by. + * + * @param {{ turns?: Array }} payload the body the tail would be cut from + * @param {number} count how many records the page holds + * @param {string} askedFingerprint the fingerprint of those records' versions + * @returns {{ status: number, code: string, message: string } | null} + */ +export function deltaRefusal(payload, count, askedFingerprint) { + const fingerprint = typeof askedFingerprint === 'string' ? askedFingerprint.trim() : ''; + if (!Number.isInteger(count) || count < 0 || !/^[0-9a-f]{16}$/.test(fingerprint)) { + return { + status: 400, + code: 'trajectory_fingerprint_required', + message: '会话已更新，请重新加载后再继续', + }; + } + if (recordSetFingerprint(payloadRecordVersions(payload, count)) !== fingerprint) { + return { + status: 409, + code: 'trajectory_version_mismatch', + message: '该记录所在的版本已变化，请重新加载会话', + }; + } + return null; +} + /** * Cheap single-pass counters used by the session list; values are approximate by design. * @@ -438,7 +807,7 @@ export function activeCatalogBytes(catalog) { * The catalog-recorded sizes are a snapshot, and the active `messages.jsonl` keeps being appended * to after the catalog was written — so "we read fewer bytes than the file now has" is not evidence * that the file is too large. Reporting it as truncation is how a merely-active session ended up - * claiming the first 64 MB when it was nowhere near it. + * claiming the whole read cap when it was nowhere near it. * * @param {number} size the file's size at read time * @param {number} limit the bytes we were willing to read @@ -633,6 +1002,31 @@ export function activeCatalogGeneration(catalog, generations) { return generations.length > 0 ? generations[generations.length - 1].generation : null; } +/** + * Whether `candidate` is `root` itself, or something under it. + * + * One definition, for every place that has to answer "is this path inside that one": the artifact + * file a catalog may name, the session directory a relative path may point at, and the data dir a + * caller may hand over. + * + * It compares the two paths rather than testing one string against the other, because a prefix test + * on the strings also matches a sibling whose name merely starts with the root's name (`D:\pkg-extra` + * against `D:\pkg`), and because it does not depend on the two paths having been spelled with the + * same letter case. + * + * @param {string} root + * @param {string} candidate + * @returns {boolean} + */ +export function isUnderDirectory(root, candidate) { + const base = resolve(root); + const target = resolve(candidate); + if (target === base) return true; + const inside = relative(base, target); + if (isAbsolute(inside)) return false; + return inside !== '..' && !inside.startsWith(`..${sep}`); +} + /** * Resolves a catalog artifact to a readable file inside the session directory. * @@ -650,7 +1044,7 @@ export function resolveArtifactPath(dir, artifact) { if (!SAFE_ARTIFACT_NAME.test(artifact.fileName) || artifact.fileName.includes('..')) return null; const root = resolve(dir); const candidate = resolve(root, artifact.active ? artifact.fileName : join('snapshots', artifact.fileName)); - if (candidate !== root && !candidate.startsWith(root + sep)) return null; + if (!isUnderDirectory(root, candidate)) return null; return candidate; } @@ -773,7 +1167,7 @@ export async function readSessionIdentity(dir) { const manifest = await readJsonIfPresent(dir, 'manifest.json'); const catalog = await readJsonIfPresent(dir, 'history-catalog.json'); const manifestId = manifest && typeof manifest.sessionId === 'string' ? manifest.sessionId.trim() : ''; - const id = manifestId || decodeSessionIdFromDirName(basename(dir)) || basename(dir); + const id = publishedSessionId(manifestId, dir); const createdAtMs = manifest && Number.isFinite(Number(manifest.createdAtMs)) ? Number(manifest.createdAtMs) : null; const generations = parseCatalogGenerations(catalog); return { @@ -785,32 +1179,69 @@ export async function readSessionIdentity(dir) { }; } +/** + * The id a directory is published under. + * + * One writer, and the same answer everywhere. The manifest beside the transcript names the + * session when it is there, and the manifest is the authority: a directory's name is a claim, + * not a fact. The name names the session when it decodes, and the folder's own name is the + * last resort -- a directory whose name does not decode and that has no manifest has no id of + * its own, and its folder name is what it can be reached by. + * + * A name that does not decode names NO session, which is not the same as disagreeing with one: + * such a directory is identified by its manifest alone, and it is still a real session. + * + * @param {string | null | undefined} manifestId + * @param {string} dir + * @returns {string} + */ +export function publishedSessionId(manifestId, dir) { + return manifestId || decodeSessionIdFromDirName(basename(dir)) || basename(dir); +} + +/** + * The session id the manifest beside the transcript names, or null when there is no manifest. + * + * A directory's name and the manifest beside its transcript are two copies of the same id. This + * reads only the manifest -- the catalogue beside it says nothing about who the directory holds + * -- and it says nothing when the manifest is absent, because a directory with no manifest is + * judged by its name alone. + * + * @param {string} dir + * @returns {Promise} + */ +async function readManifestSessionId(dir) { + const manifest = await readJsonIfPresent(dir, 'manifest.json'); + const id = manifest && typeof manifest.sessionId === 'string' ? manifest.sessionId.trim() : ''; + return id || null; +} + /** * The whole candidate set from one walk, as two products of the same traversal. * * These used to be one thing, and that is the defect. `slice(0, MAX_SESSION_LIST)` ran last, - * so the only handle the service had on a session directory was "one of the 30 most recently + * so the only handle the service had on a session directory was "one of the most recently * touched" — and `resolveTrajectoryRequest` resolved an explicit id by scanning exactly those - * 30. Measured on this machine: 146 candidate directories, 30 exposed, 116 of them - * unreachable by id (window inside 200/200, a sample outside 404/404). A session the user - * could name but not open is a correctness bug, not a small one. + * MAX_SESSION_LIST. A session the user could name but not open is a correctness bug, not a small one. * * So the traversal now yields two things: * * - `recentEntries` — sorted by mtime, capped at `MAX_SESSION_LIST`. This is what the picker * shows, and it stays exactly as small as it was. - * - `byDecodedId` — every candidate directory, keyed by the id decoded from its own name. - * Decoding reads no file: the walk already stat'ed `messages.jsonl` for every candidate, so - * the index costs no I/O the walk was not doing anyway. + * - `bySessionId` — every candidate directory, keyed by the id that directory is published + * under. The walk decides that id once, when it reads the manifest it already reads to decide + * whether the directory is publishable at all, so the index costs no I/O the walk was not + * doing anyway -- and every id the list shows is an id this index can resolve. * * Two properties this must not lose: * - * - A decoded name is a *claim*, not a fact. `readSessionIdentity` prefers `manifest.sessionId` - * over the directory name, so every caller of this index still reads the identity of the one - * directory it picked and checks the id before serving it. - * - Two directories can decode to the same id. `Map.set` would silently keep whichever came - * last, which makes the answer depend on `readdir` order. Duplicates are therefore kept as a - * list and resolved against the manifests, not discarded and not guessed. + * - The id is a *decision*, not a guess. The walk refuses a directory whose name and manifest + * disagree, so a directory in this index is one session's directory; and every caller still + * reads the identity of the one directory it picked and checks the id before serving it, + * because the manifest can change under the walk. + * - Two directories can hold the same id. `Map.set` would silently keep whichever came last, + * which makes the answer depend on `readdir` order. Duplicates are therefore kept as a list + * and resolved against the manifests, not discarded and not guessed. * * `truncated` says the walk hit `MAX_SESSION_CANDIDATES` and stopped early. That budget is a * resource guard, and it must not be allowed to masquerade as "this session does not exist" — @@ -818,8 +1249,9 @@ export async function readSessionIdentity(dir) { * truthful one. * * @param {string} sessionsRoot - * @returns {Promise<{ recentEntries: Array<{ dir: string, sizeBytes: number, lastActiveAt: number }>, - * byDecodedId: Map>, + * @returns {Promise<{ recentEntries: Array<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>, + * allEntries: Array<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>, + * bySessionId: Map>, * truncated: boolean }>} */ export async function collectSessionCandidates(sessionsRoot) { @@ -853,21 +1285,56 @@ export async function collectSessionCandidates(sessionsRoot) { } }; await walk(sessionsRoot, 0); - entries.sort((a, b) => b.lastActiveAt - a.lastActiveAt); - /** @type {Map>} */ - const byDecodedId = new Map(); - for (const entry of entries) { - const id = decodeSessionIdFromDirName(basename(entry.dir)); - // A name that does not decode is still a real session; it just has no fast path and is - // found the slow way. Dropping it here would turn "no fast path" into "does not exist". - if (!id) continue; - const bucket = byDecodedId.get(id); + // A directory is a session's directory only when its name and the manifest beside its + // transcript name that session: they are two copies of the same id, and the rule the recorded + // path and the 'latest' binding already follow -- the name AND the manifest -- is the one that + // belongs here too. + // + // A directory where the two disagree is nobody's. Publishing it would put one session's id on + // another session's transcript, and the row would mix the manifest's id with the name's + // metadata -- two sources in one row. So it does not enter the walk: the list does not publish + // it, the validator does not see it, and the trajectory route does not serve it. + // + // And the id it claims is not among the published ones, so it is not a parent the reader can + // pick -- a run hanging off it becomes a conversation of its own, by the rule that a parent the + // reader cannot reach is not a parent. + // + // The id every directory is PUBLISHED under is decided here, once. Everything downstream -- + // the title lookup, the layer, the row, the index the route resolves an id through -- reads it + // off the entry instead of decoding the name again, because a name that does not decode names + // no session at all: the manifest is then the only identity that directory has, and a parent + // identified that way still holds its runs, still has its title, and is still openable by the + // id the list shows. + const judged = await Promise.all(entries.map(async (entry) => { + const claimed = decodeSessionIdFromDirName(basename(entry.dir)); + const manifestId = await readManifestSessionId(entry.dir); + // A name that does not decode names no session, so there is nothing for the manifest to + // disagree with: the manifest is the only identity there is. + const holds = !claimed || !manifestId || manifestId === claimed; + return { entry, holds, sessionId: publishedSessionId(manifestId, entry.dir) }; + })); + const publishable = judged.filter((one) => one.holds) + .map((one) => ({ ...one.entry, sessionId: one.sessionId })); + publishable.sort((a, b) => b.lastActiveAt - a.lastActiveAt); + + /** @type {Map>} */ + const bySessionId = new Map(); + for (const entry of publishable) { + // The key is the id the entry is published under, so every id the list shows is an id this + // index can resolve. A name that does not decode names no session, and the folder name is + // what that directory can be reached by -- so it is in here under that name rather than + // missing, which would turn "no fast path" into "does not exist". + const bucket = bySessionId.get(entry.sessionId); if (bucket) bucket.push(entry); - else byDecodedId.set(id, [entry]); + else bySessionId.set(entry.sessionId, [entry]); } - return { recentEntries: entries.slice(0, MAX_SESSION_LIST), byDecodedId, truncated }; + // `allEntries` is the same walk, uncut. The capped window is what 'latest' falls back to and + // what the picker used to draw from; the list of MAIN sessions needs a deeper pool than that, + // because a sub-agent sits in that window and would otherwise take a main session' place. + // The walk already stat'ed every one of these, so the deeper pool costs no extra disk work. + return { recentEntries: publishable.slice(0, MAX_SESSION_LIST), allEntries: publishable, bySessionId, truncated }; } /** @@ -918,7 +1385,7 @@ function toNumber(value) { } /** - * Resolve the conversation the user is actually in. + * Resolve the conversation the runtime database reports as current. * * The Host never tells the page which session opened it, so we ask the runtime database * instead. Ranking, strongest signal first: @@ -926,6 +1393,11 @@ function toNumber(value) { * 2. status = 'started' — open and not finished * 3. most recently updated — the fallback for a long-idle conversation * + * This is what the runtime database says, and the app infers from it: a ranking is a best + * answer, not a certainty, and two conversations holding a lease at once are reported as + * ambiguous rather than resolved here. Nothing in this file can see which conversation the + * reader is actually looking at. + * * Only real user conversations qualify. `session_kind = 'conversation'` with a NULL * `purpose` excludes cron runs (`purpose = 'cron:...'`) and background worker/explore * tasks (`purpose = 'local-background-task:bg_...'`), which are exactly the sessions that @@ -988,18 +1460,58 @@ export async function resolveCurrentConversation(home) { } catch { return null; } finally { - // Every call opened its own handle and none of them closed it. Polling every 2.5s meant + // Every call opened its own handle and none of them closed it. Polling on every tick meant // 20–30 live handles waiting on the GC, and dispose() left them open too. closeQuietly(db); } } +/** + * The columns the picker reads, widest first. + * + * A runtime database written before one of these columns existed does not have it, and preparing a + * statement over a column that is not there throws before a single row has been read. So the read + * walks down this list until one statement prepares: each rung drops the newest column and keeps + * every older one, which is what lets an old database still yield titles, kinds and parents + * instead of nothing at all. + */ +const SESSION_TITLE_SELECTS = [ + 'select session_id, title, session_kind, parent_session_id, purpose, workspace_dir from local_runtime_sessions where session_id = ?', + 'select session_id, title, session_kind, parent_session_id, purpose from local_runtime_sessions where session_id = ?', + 'select session_id, title, session_kind, parent_session_id from local_runtime_sessions where session_id = ?', + 'select session_id, title, session_kind from local_runtime_sessions where session_id = ?', +]; + /** * Titles for the session picker. Missing database simply means the caller keeps its * filesystem-derived label. + * + * Five facts come back per session, and the last three are what the picker needs to stop + * presenting every session as if it were the reader's own conversation: + * + * `title` what the session is called + * `kind` the shape the runtime gave it + * `parent` the session it was spawned from, when it was spawned from one + * `purpose` what the session is FOR -- a cron run, a background task, a lightweight exchange + * `workspace` the project directory it belongs to + * + * The parent id is the fact that a branch exists, and it is the ONLY part of that + * relationship that is always available: the list is the most recent N sessions, and a child + * is always newer than the parent that spawned it, so the parent's title is routinely outside + * the window. Carrying the id and looking the title up would therefore be a lookup that fails + * exactly when a reader is most likely to want it. + * + * Asking for a column is the optional part, and it has to be optional: a runtime database + * written before child sessions existed has no `parent_session_id`, and preparing a statement + * over a column that is not there throws before a single row has been read. Under the empty + * catch below that is not a degraded list -- it is every title and every kind in the page + * silently reverting to the first sentence of the transcript, which is what a reader would see + * as "the titles broke and nothing says why". The mapping below already answers `null` for a + * row that carries no parent, so the rows the lower rungs return need no change at all. + * * @param {string} home * @param {readonly string[]} sessionIds - * @returns {Promise>} + * @returns {Promise>} */ export async function readSessionTitles(home, sessionIds) { const result = new Map(); @@ -1007,38 +1519,43 @@ export async function readSessionTitles(home, sessionIds) { if (!db) return result; try { if (sessionIds.length === 0) return result; - // The parent id is the fact that a branch exists, and it is the ONLY part of that - // relationship that is always available: the list is the most recent N sessions, and a child - // is always newer than the parent that spawned it, so the parent's title is routinely outside - // the window. Carrying the id and looking the title up would therefore be a lookup that fails - // exactly when a reader is most likely to want it. - // - // Asking for the column is the optional part, and it has to be optional: a runtime database - // written before child sessions existed has no `parent_session_id`, and preparing a statement - // over a column that is not there throws before a single row has been read. Under the empty - // catch below that is not a degraded list -- it is every title and every kind in the page - // silently reverting to the first sentence of the transcript, which is what a reader would see - // as "the titles broke and nothing says why". The mapping below already answers `null` for a - // row that carries no parent, so the row this fallback returns needs no change at all. - let stmt; - try { - stmt = db.prepare('select session_id, title, session_kind, parent_session_id from local_runtime_sessions where session_id = ?'); - } catch { + let stmt = null; + for (const sql of SESSION_TITLE_SELECTS) { try { - stmt = db.prepare('select session_id, title, session_kind from local_runtime_sessions where session_id = ?'); + stmt = db.prepare(sql); + break; } catch { - return result; + // This database does not have that column yet; ask for less. } } + if (!stmt) return result; for (const id of sessionIds) { const row = stmt.get(id); if (!row) continue; + const parent = typeof row.parent_session_id === 'string' && row.parent_session_id !== '' + ? row.parent_session_id + : null; + // The parent column is a foreign key into this very table: it names the session that + // spawned this one. A value that names no session here is not a parent, and publishing it + // would put a dangling link on the wire -- and the page groups its runs by that value, so + // a name that is not a session's id is a name the page has no business keying a group by. + // + // The lookup is the statement that read this very row, so there is no second query that + // could fail to prepare, and therefore no path on which a parent is published without + // having been looked up: a database written before child sessions existed has no + // `parent_session_id` column, so every parent above is null and there is nothing to + // look up at all. + // + // A lookup that throws takes this row out of the map along with everything else, and a + // row that is not in the map has no parent either. So the promise holds on every path: + // a parent that is published names a session. + const parentNames = parent ? Boolean(stmt.get(parent)) : false; result.set(id, { title: typeof row.title === 'string' && row.title.trim() ? row.title.trim() : null, kind: typeof row.session_kind === 'string' ? row.session_kind : null, - parent: typeof row.parent_session_id === 'string' && row.parent_session_id !== '' - ? row.parent_session_id - : null, + parent: parentNames ? parent : null, + purpose: typeof row.purpose === 'string' && row.purpose.trim() ? row.purpose.trim() : null, + workspace: typeof row.workspace_dir === 'string' && row.workspace_dir.trim() ? row.workspace_dir.trim() : null, }); } } catch { @@ -1049,6 +1566,454 @@ export async function readSessionTitles(home, sessionIds) { return result; } +/** + * What kind of session this is, as a closed set of tokens the page can render. + * + * This is the only place that knows what a runtime `purpose` means. The raw value is free text -- + * it carries whatever a person wrote into it -- so it is never published; what travels is the token + * derived from it. + * + * `purpose` answers first because it is the more specific fact: a session can be a cron run and + * still be recorded with the kind of a conversation. `kind` is the fallback, for a database whose + * `purpose` column does not exist yet. + * + * @param {string | null | undefined} kind + * @param {string | null | undefined} purpose + * @returns {'lightweight' | 'scheduled' | 'task' | null} + */ +export function sessionTagFor(kind, purpose) { + const p = typeof purpose === 'string' ? purpose.trim() : ''; + const k = typeof kind === 'string' ? kind.trim().toLowerCase() : ''; + if (p === 'context-mode:lightweight') return 'lightweight'; + if (p.startsWith('cron:') || k === 'cron') return 'scheduled'; + if (p.startsWith('local-background-task:') || p.startsWith('local-task:') || k === 'task') return 'task'; + return null; +} + +/** + * @param {string | null | undefined} value + * @returns {{ value: string, windows: boolean } | null} + */ +function normaliseWorkspacePath(value) { + if (typeof value !== 'string') return null; + const trimmed = value.trim(); + if (!trimmed) return null; + const slashed = trimmed.replace(/\\/g, '/'); + const withoutTail = slashed.length > 1 ? slashed.replace(/\/+$/, '') : slashed; + // A drive letter or a UNC root is what makes a path a Windows path, wherever the comparison runs. + const windows = /^[A-Za-z]:\//.test(withoutTail) || withoutTail.startsWith('//'); + return { value: withoutTail, windows }; +} + +/** + * Whether two workspace paths name the same directory. + * + * The value is an absolute path on the reader's machine, so it is compared here and never sent + * anywhere: the page receives an ordered list, and a path in it would be a location on somebody's + * disk. Two spellings of one path have to compare equal, because the same workspace is written + * with backslashes in one row and with forward slashes in another, and a Windows path does not + * distinguish `D:\work` from `D:\WORK`. + * + * Case is folded only for Windows-shaped paths. A POSIX path is compared as it stands, because + * `/work/Project` and `/work/project` really are two directories there. + * + * @param {string | null | undefined} left + * @param {string | null | undefined} right + * @returns {boolean} + */ +export function sameWorkspacePath(left, right) { + const a = normaliseWorkspacePath(left); + const b = normaliseWorkspacePath(right); + if (!a || !b) return false; + if (a.windows || b.windows) return a.value.toLowerCase() === b.value.toLowerCase(); + return a.value === b.value; +} + +/** + * The workspace a session belongs to, as the picker reads it: null when the database has no such + * column, no such row, or no value in it. + * + * @param {Map} titles + * @param {string | null | undefined} sessionId + * @returns {string | null} + */ +function workspaceOfSession(titles, sessionId) { + if (!sessionId || !(titles instanceof Map)) return null; + const meta = titles.get(sessionId); + if (!meta || typeof meta.workspace !== 'string' || !meta.workspace.trim()) return null; + return meta.workspace.trim(); +} + +/** + * Build a session entry from a recorded relative directory. + * + * The value is a column the runtime wrote rather than a name this app chose, so it earns the same + * containment check `resolveArtifactPath` gives catalog file names: a path that resolves outside + * the sessions root is refused, not clamped. The picker and the 'latest' resolver share this one + * function, so a second copy of the check cannot drift away from the first. + * + * The containment check alone is not enough, and the gap is the same shape as the one it closes: + * a path can sit INSIDE the sessions root and still belong to a different session than the one + * that recorded it. Then the caller holds one session's directory under another session's id — + * the picker would show the current conversation's id twice and drop the conversation itself, and + * the 'latest' resolver would bind one session's id to another session's directory. So the + * session the path is supposed to be is named here, and a path that turns out to be a different + * session's is refused exactly the way an outside path is. + * + * @param {string} sessionsRoot + * @param {string | null | undefined} relativeDir + * @param {string | null | undefined} [expectedSessionId] + * @returns {Promise<{ entry: { dir: string, sizeBytes: number, lastActiveAt: number } | null, rejected: boolean }>} + */ +export async function entryFromRelativeDir(sessionsRoot, relativeDir, expectedSessionId) { + if (typeof relativeDir !== 'string' || !relativeDir.trim()) return { entry: null, rejected: false }; + const root = resolve(sessionsRoot); + const dir = resolve(root, relativeDir.trim().replace(/\\/g, '/')); + if (!isUnderDirectory(root, dir)) return { entry: null, rejected: true }; + try { + const fileStat = await stat(join(dir, 'messages.jsonl')); + // The id of the session this directory holds. It travels with the entry, like it does for + // the walk's own rows: a row built from a recorded path is published under an id too, and a + // row with no id is a row the page cannot group, label or select. + const identity = await readSessionIdentity(dir); + if (expectedSessionId) { + // The directory name is not the only place the session id is written down, and it is not + // the authority either. The service reads a session's identity from the manifest the + // runtime wrote beside its transcript (`readSessionIdentity`), and the manifest is what + // says which session this directory holds. A directory whose name encodes one session and + // whose manifest names another is the OTHER session's directory, whatever the folder is + // called. + // + // So both have to be the session this path is supposed to be, and they are checked one at + // a time because they can disagree in both directions. The name has to be it: that is the + // check that keeps one directory from being listed twice under two ids. And the manifest + // has to be it too: a manifest naming another session is a recorded path pointing at + // somebody else's transcript, and accepting it would put the current conversation's id on + // another session's messages -- the same defect the name check exists for, one column + // further in. + // + // A name that does not decode names NO session, and that is not the same as disagreeing + // with one: there is nothing for the manifest to disagree with, so the manifest decides + // alone. A directory with no manifest is judged by its name alone, which is the case that + // was always here. + const claimed = decodeSessionIdFromDirName(basename(dir)); + if (claimed && claimed !== expectedSessionId) { + return { entry: null, rejected: true }; + } + if (identity.id !== expectedSessionId) { + return { entry: null, rejected: true }; + } + } + return { + entry: { + dir, + sizeBytes: fileStat.size, + lastActiveAt: Math.floor(fileStat.mtimeMs), + sessionId: identity.id, + }, + rejected: false, + }; + } catch { + return { entry: null, rejected: false }; + } +} + +/** + * The entry that holds a session: found by the id the walk published it under, and confirmed by + * the manifest beside its transcript. + * + * The id is the walk's decision, and the walk refuses a directory whose name and manifest + * disagree -- so a candidate in the index holds the session it is indexed under. The manifest is + * read again anyway, because it can change under the walk, and a directory that has stopped + * holding that session must not be served as it. + * + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} entries + * @param {string} sessionId + * @returns {Promise<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string } | null>} + */ +export async function verifiedEntryForSessionId(entries, sessionId) { + if (!sessionId) return null; + for (const candidate of entries) { + if (candidate.sessionId !== sessionId) continue; + const identity = await readSessionIdentity(candidate.dir); + if (identity.id === sessionId) return candidate; + } + return null; +} + +/** + * The newest candidate that still holds the session it was published under. + * + * The walk's admission is a decision made when the walk ran, and the manifest beside a + * transcript can change under it. The explicit-id routes ask again before they serve a + * candidate -- the fast path re-reads the identity, the slow path asks the same question the + * walk asks -- so a directory whose manifest no longer agrees with its name is refused by + * both. The `latest` fallback used to take the first entry of the window as it stood, which + * served exactly that directory: the same transcript, under the id its manifest now claims, + * while every explicit id for it was refused. One rule, one question, three places. + * + * The question is the one the id itself answers: the id the directory holds now -- the + * manifest's, or the name's when the manifest is gone -- has to be the id it was published + * under. A manifest that changes from one id to another moves that answer and nothing else, + * which is exactly why it has to be asked again rather than read off the walk. + * + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} entries + * @returns {Promise<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string } | null>} + */ +export async function firstHoldingEntry(entries) { + for (const candidate of entries) { + const identity = await readSessionIdentity(candidate.dir); + if (identity.id === candidate.sessionId) return candidate; + } + return null; +} + +/** + * The visible set the picker shows, and the order it is shown in. + * + * Two sets live here and they are not the same set. `entries` is what the caller handed over: + * the sessions the visible set is drawn from, mtime-descending. The list route hands over the + * walk's conversations -- the runs were split off before this was called, so a run is never in + * here. The visible set is what the READER may pick from, and it is allowed to differ. + * + * The order is: the conversation the runtime database reports as current, then the sessions of + * that conversation's workspace, then everything else -- each group mtime-descending. "The + * current workspace" means the `workspace_dir` the runtime database records for that + * conversation, and only that; the path itself never leaves this function. + * + * "The current conversation" is what the runtime database reports, not what the reader is in: + * the runtime knows which session it last updated, and this app cannot see any more than that. + * The database's answer is a ranking (a live lease, then an open turn, then the most recently + * updated), and `resolveCurrentConversation` reports the ambiguity when two conversations hold + * a lease at once rather than picking one silently. + * + * The current conversation is promoted only when it IS a conversation. The layer is decided by + * the parent column and the chain it forms, and `splitSessionLayers` has already cleared the + * parent of every session it put in the conversation layer -- so a session that still has a + * parent here is a run, and a run that wins the current-conversation resolution is not + * promoted back into the conversation layer. It is already in the run layer, where it belongs, + * and promoting it here would put one session in both controls at once: the conversation pick + * would offer a run, and the run pick would offer the same run again. + * + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} entries + * @param {{ sessionId: string, relativeDir?: string | null } | null} current + * @param {Map} titles + * @param {string} sessionsRoot + * @param {number} limit + * @param {() => void} [onReject] + * @returns {Promise>} + */ +export async function buildVisibleSessionList(entries, current, titles, sessionsRoot, limit, onReject) { + if (!current || typeof current.sessionId !== 'string' || !current.sessionId) { + // No runtime database, or no current conversation: the visible set is exactly what the walk + // found, in the walk's order. This is what the picker did before the current conversation was + // pinned into it, and it is still what it does when there is nothing to pin. + return entries.slice(0, limit); + } + const currentId = current.sessionId; + // The layer is decided by the parent column and the chain it forms, and the picker has two + // of them. The split has already cleared the parent of every session it put in the + // conversation layer, so a session that still has a parent is a run -- and a run that wins + // the current-conversation resolution is not promoted into the conversation layer. Promoting + // it would put one session in both controls at once. + const currentMeta = titles instanceof Map ? titles.get(currentId) : null; + const currentParent = currentMeta && typeof currentMeta.parent === 'string' && currentMeta.parent + ? currentMeta.parent + : null; + let currentEntry = null; + if (!currentParent) { + // The current conversation's own row, found among the walk's rows and CONFIRMED by the + // manifest beside the transcript: the directory name is only a claim. A directory whose name + // says this session and whose manifest says another one is that other session's directory, + // and promoting it would put this session's id on somebody else's messages. + currentEntry = await verifiedEntryForSessionId(entries, currentId); + if (!currentEntry && current.relativeDir) { + // Not in the walk -- build the entry straight from the recorded path. The session it + // is supposed to be is named, so a path that turns out to be a different session's directory + // is refused instead of being shown under this id. + const built = await entryFromRelativeDir(sessionsRoot, current.relativeDir, currentId); + if (built.rejected && typeof onReject === 'function') onReject(); + currentEntry = built.entry; + } + } + const currentWorkspace = workspaceOfSession(titles, currentId); + const rest = entries.filter((entry) => entry !== currentEntry); + const same = []; + const other = []; + for (const entry of rest) { + const workspace = workspaceOfSession(titles, entry.sessionId); + if (currentWorkspace !== null && sameWorkspacePath(workspace, currentWorkspace)) same.push(entry); + else other.push(entry); + } + // Both groups keep the walk's order, which is mtime-descending. Stated rather than inherited: + // the walk's order is an implementation detail of the traversal, and the picker promises one. + const byRecency = (a, b) => b.lastActiveAt - a.lastActiveAt; + same.sort(byRecency); + other.sort(byRecency); + const ordered = currentEntry ? [currentEntry, ...same, ...other] : [...same, ...other]; + return ordered.slice(0, limit); +} + +/** + * The two layers of the picker: the main sessions, and the sub-agents hanging off each of them. + * + * A sub-agent is a session like any other on disk — its own directory, its own `messages.jsonl` — + * and the runtime hangs it off its parent with one column. So the layer is a fact about a row, + * and it is read from that column and from the chain that column forms. `purpose` is not the + * judge here: it says what a session IS (a scheduled run, a background task), which is a different + * question from what it hangs off, and the two are orthogonal — a sub-agent is a task AND somebody's + * child. + * + * The parent's id and the child's id are the same kind of id. Measured on this machine: all 208 + * session directories carry a `manifest.json` whose `sessionId` equals the id decoded from the + * directory name, and that is the id the runtime database stores. So the parent's id here is the + * id the page will see on the parent's row, and the client can join the two layers on it. + * + * The column alone is not enough to decide the layer, though, and the cases where it is not + * enough are the ones where a session disappears from the picker. The column answers "does this + * session have a parent". The layer is "is that parent a CONVERSATION", and that is a question + * about the chain: + * + * - A cycle — A's parent is B, B's parent is A — has no conversation at the end of it. Read the + * column alone, both of them are runs, of each other, so neither is a conversation, and the + * conversation list comes back with neither in it: they are in neither control, and the + * reader cannot reach either one. A cycle is dissolved instead. Both are conversations, and + * their parent columns are dropped, because a conversation that names a parent names a run. + * - A chain that leaves the sessions this payload knows about — the parent has a row in the + * runtime database but no directory in the pool — has no conversation to point at either: the + * conversation a reader would pick to see this run is not in the list. The run is a + * conversation itself, so that it is somewhere the reader can reach. + * - A chain longer than one step — the parent is itself a run — puts the run in the + * conversation layer, because the two controls are one conversation and the runs of THAT one. + * + * So the rule is one sentence: a session is a run iff its parent is a conversation, and a + * conversation is a session with no parent, or one whose parent is not a conversation. The two + * halves are given together on purpose. One half alone has no answer for a cycle, and the answer + * "the column says so" is the one that loses both sessions. + * + * The map is corrected in place: a session this puts in the conversation layer has its parent + * cleared, so the payload, the validator and the current-conversation promotion all read the same + * value. A copy would leave one of the three reading the uncorrected one, and the body and the + * ETag would then describe two different lists. + * + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} entries + * @param {Map} titles + * @returns {{ mains: Array<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>, childrenByParent: Map> }} + */ +export function splitSessionLayers(entries, titles) { + const mains = []; + const childrenByParent = new Map(); + const map = titles instanceof Map ? titles : new Map(); + // The sessions this payload knows about: the ones with a directory in the walk. NOT the ones + // with a row in the runtime database -- the map holds sessions that have no transcript here, + // the current conversation among them, and a parent that is only a row is not something the + // reader can pick, so a run hanging off it would be a run the reader cannot reach. The rule is + // about what the reader can reach, so the membership test is about the walk. + // + // The membership is the id each directory is PUBLISHED under, which is not always the id its + // name decodes to: a name that does not decode names no session, and the manifest beside the + // transcript is then the only identity that directory has. A parent identified that way is in + // the list the reader can pick, so the runs hanging off it are its runs. + const known = new Set(); + for (const entry of entries) { + if (entry.sessionId) known.add(entry.sessionId); + } + + /** + * The session a session hangs off, or null when it hangs off nothing. + * @param {string} id + * @returns {string | null} + */ + const parentOf = (id) => { + const meta = map.get(id); + if (!meta) return null; + const declared = typeof meta.parent === 'string' && meta.parent ? meta.parent : null; + return declared; + }; + + // The sessions that sit on a cycle: following the parent links from one of them comes back to + // it. Every link is followed through the same map the layer is decided by, so a session on a + // cycle is one this pass reaches -- and the walk below cannot run forever, because it stops at + // the first of these. + const onCycle = new Set(); + for (const id of map.keys()) { + const path = []; + const at = new Map(); + let cur = id; + while (cur && !at.has(cur)) { + at.set(cur, path.length); + path.push(cur); + cur = parentOf(cur); + } + if (cur && at.has(cur)) { + for (let i = at.get(cur); i < path.length; i += 1) onCycle.add(path[i]); + } + } + + // The layer of every session: a conversation, or a run of one. Walking up the chain flips the + // layer at every step, because a run's parent is a conversation and a conversation's parent is + // not one. The walk ends where the chain stops making sense: at a session with no parent, at a + // parent this payload does not know, or at a cycle. + const CONVERSATION = 'conversation'; + const RUN = 'run'; + const layerOf = new Map(); + const flip = (layer) => (layer === CONVERSATION ? RUN : CONVERSATION); + + /** + * @param {string} start + * @returns {string} + */ + const classify = (start) => { + if (layerOf.has(start)) return /** @type {string} */ (layerOf.get(start)); + const chain = []; + let cur = start; + let base; + while (true) { + if (layerOf.has(cur)) { base = flip(/** @type {string} */ (layerOf.get(cur))); break; } + chain.push(cur); + if (onCycle.has(cur)) { base = CONVERSATION; break; } + const parent = parentOf(cur); + if (!parent || !known.has(parent)) { base = CONVERSATION; break; } + cur = parent; + } + let layer = base; + for (let i = chain.length - 1; i >= 0; i -= 1) { + layerOf.set(chain[i], layer); + layer = flip(layer); + } + return /** @type {string} */ (layerOf.get(start)); + }; + + for (const id of map.keys()) classify(id); + + // The correction, and the reason the layer can be read off the parent column again: a session + // in the conversation layer has no parent to name, so its parent column is cleared here, once, + // for everything that reads it afterwards. The body, the validator and the current-conversation + // promotion all read this same map, so all three say the same thing about the same row. + for (const [id, meta] of map) { + if (!meta) continue; + if (layerOf.get(id) === CONVERSATION && typeof meta.parent === 'string' && meta.parent) { + meta.parent = null; + } + } + + for (const entry of entries) { + const id = entry.sessionId; + const parent = id ? parentOf(id) : null; + if (!parent) { + mains.push(entry); + continue; + } + const bucket = childrenByParent.get(parent); + if (bucket) bucket.push(entry); + else childrenByParent.set(parent, [entry]); + } + // Most recently active first, inside every bucket. The list says so in the one place that + // orders it, so the order cannot be one thing on the server and another on the page. + for (const bucket of childrenByParent.values()) bucket.sort((a, b) => b.lastActiveAt - a.lastActiveAt); + return { mains, childrenByParent }; +} + /** * Closing a read-only handle cannot fail in a way the caller can act on — the work it was * opened for is already done or already abandoned. Swallowing keeps every call site from @@ -1188,7 +2153,7 @@ export function toolCallSummary(name, args) { * The message itself was carried on every row and taken off again by this function. That is * what kept the bulk response small, and it stays true. What did not survive scrutiny is how * the size was decided: buildRawPayload() had to JSON.stringify the message to measure it, so - * every rebuild serialized about 100 MB of message bodies to compare them against a 64 KB cap + * every rebuild serialized about 100 MB of message bodies to compare them against the read cap * and then threw the string away — for a body the page reads for one record at a time, if at * all. Pinned by tests/verify-raw-lazy.mjs. * @@ -1282,10 +2247,10 @@ export function slimPayloadForWire(payload) { * @param {{ turns: Array }} payload * @param {string} needle already normalised and folded * @param {number} limit - * @returns {{ hits: Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null }>, total: number }} + * @returns {{ hits: Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null, v: string }>, total: number }} */ export function collectSearchHits(payload, needle, limit) { - /** @type {Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null }>} */ + /** @type {Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null, v: string }>} */ const hits = []; let total = 0; const turns = payload && Array.isArray(payload.turns) ? payload.turns : []; @@ -1303,6 +2268,10 @@ export function collectSearchHits(payload, needle, limit) { // without opening every row the needle found. summary: typeof record.summary === 'string' ? record.summary : '', generation: typeof record.generation === 'number' ? record.generation : null, + // The version of the record this hit names. The index alone is a position, and the position + // is only the same record in another body if the version says so — the page checks it + // against the body it is showing before it lets the hit filter anything. + v: typeof record.v === 'string' ? record.v : '', }); } } @@ -2415,18 +3384,29 @@ export function etagMatches(header, etag) { * SESSION_CACHE_TTL_MS to the descriptor cache this reuses, so the bound is that cache's and not * a new one. tests/verify-list-cache.mjs pins the TTL relationship that says so. * - * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number }>} visible + * The sub-agents are in it for the same reason the kind badge is: they are on the page, and a + * sub-agent that starts, ends, or is renamed changes what the page says. A material that did not + * move would answer 304 for a body the caller never held — and the body it never held is the one + * with the new sub-agent in it. + * + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} visible * @param {Map} titles * @param {number} limit + * @param {ReadonlyArray<{ dir: string, sizeBytes: number, lastActiveAt: number, sessionId: string }>} [children] * @returns {string} */ -export function sessionsValidator(visible, titles, limit) { +export function sessionsValidator(visible, titles, limit, children) { const parts = ['limit=' + limit]; for (const entry of visible) { - const meta = titles.get(decodeSessionIdFromDirName(basename(entry.dir))); + const meta = titles.get(entry.sessionId); const title = meta ? String(meta.title ?? "") : ""; const kind = meta ? String(meta.kind ?? "") : ""; const parent = meta ? String(meta.parent ?? "") : ""; + // The kind badge the page draws is derived from the row's kind and purpose, so it is + // reader-visible on its own and belongs in the material for the same reason `kind` does. A + // runtime database that changes a session's purpose changes what the row says, and a tag that + // did not move would answer 304 for a body the caller never held. + const tag = meta ? String(sessionTagFor(meta.kind, meta.purpose) ?? "") : ""; // Injective by construction, which no delimiter is. Any character can occur inside a SQLite TEXT // column — including the NUL this used to join on — so a title of "a\0b" beside a kind of "c" and // a title of "a" beside a kind of "b\0c" produced byte-identical material: one ETag for two rows @@ -2434,7 +3414,25 @@ export function sessionsValidator(visible, titles, limit) { // different separator cannot fix that, because no character is excluded from a TEXT value. // JSON escapes the ambiguous characters, so the tuple's boundaries survive anything the store // can return, and no field needs a second representation to be safe. - parts.push(JSON.stringify([entry.dir, entry.sizeBytes, entry.lastActiveAt, title, kind, parent])); + // + // The published id is in the tuple because it is in the payload: the row's id IS the id the + // walk published this directory under, and the body the reader receives says it. A manifest + // that changes from one id to another moves the published id and moves nothing else here -- + // the directory, the transcript, the runtime row and the mtime are all the same -- so without + // this field the validator would answer 304 for a body that names a session the reader is no + // longer looking at, and the reused body would keep publishing the old id indefinitely. + parts.push(JSON.stringify([entry.sessionId, entry.dir, entry.sizeBytes, entry.lastActiveAt, title, kind, parent, tag])); + } + // The sub-agents, in the order the payload carries them: most recently active first inside + // each group. The parent id is in the tuple because the grouping is part of what the page + // shows — the same sub-agent under a different main is a different page. + for (const entry of children || []) { + const meta = titles.get(entry.sessionId); + const title = meta ? String(meta.title ?? "") : ""; + const kind = meta ? String(meta.kind ?? "") : ""; + const parent = meta ? String(meta.parent ?? "") : ""; + const tag = meta ? String(sessionTagFor(meta.kind, meta.purpose) ?? "") : ""; + parts.push(JSON.stringify([entry.sessionId, parent, entry.dir, entry.sizeBytes, entry.lastActiveAt, title, kind, tag])); } return parts.join("\n"); } @@ -2461,8 +3459,15 @@ export function trajectoryCacheFamily(dir, filter) { * The cache key for one rendered trajectory. Every component comes from a `stat`, a small * manifest/catalog file, or the already-resolved session binding — never from reading * `messages.jsonl`, which is the whole point: a matching key proves the file did not grow. + * + * The identity the directory holds is in it, because the body says which session it is about. + * A manifest that changes from one id to another leaves the directory, the transcript, the + * lineage and the mtime exactly as they were, so a key made of those would match and the + * cached body of the OLD id would be served for the NEW one — a body whose `session.id` names + * a session nobody asked for. The key has to cover the same facts the body carries. + * * @param {{ dir: string, sizeBytes: number, lastActiveAt: number }} entry - * @param {{ catalogBytes?: number | null }} identity + * @param {{ id?: string | null, catalogBytes?: number | null }} identity * @param {any} binding * @returns {string} */ @@ -2484,7 +3489,7 @@ export function trajectoryCacheKey(entry, identity, binding, filter) { .join(','); // The requested slice is part of the answer too: the same session stitched and unfiltered is // two different bodies and must never share an entry. - return `${trajectoryCacheFamily(entry.dir, filter)}${lineage}|${entry.sizeBytes}|${entry.lastActiveAt}|${identity.catalogBytes ?? ''}|${bind}`; + return `${trajectoryCacheFamily(entry.dir, filter)}${identity.id ?? ''}|${lineage}|${entry.sizeBytes}|${entry.lastActiveAt}|${identity.catalogBytes ?? ''}|${bind}`; } /** * The `?since=` cursor, or null when there is not a usable one. @@ -2634,21 +3639,111 @@ export function parsedRowsKey(filePath, size, mtimeMs, limit) { } /** - * LRU over parsed generation rows, bounded by bytes. + * LRU over parsed generation rows, bounded by bytes and by how long they are wanted. * * Bounded by bytes rather than entries because entries say nothing about volume: one * generation file is 40 KB and the next is 10 MB. A single entry larger than the whole budget * is not stored at all — it would evict everything else and still miss every time. * + * The accounting unit is the SOURCE bytes the entry was parsed from, times the heap proxy — not + * resident bytes, which V8 cannot report. That is what makes the coverage computable: the source + * bytes a round does NOT have to parse again are the lineage's source bytes minus the source + * bytes this cache holds, and the cache can say the second number exactly. + * + * Bounded by time as well, because the rows are only worth holding while somebody asks for them: + * a reader who closes the tab stops asking, and a cache with no expiry would keep that + * conversation's rows for as long as the runtime lives. The expiry is a MISS and not a deletion + * of something still wanted — the next request parses the file again and the cache fills itself + * back up. + * + * A time limit of zero means NO time limit, and it means that in one place only: the rows never + * expire and the sweeper never runs. (The other reading — "everything expires at once" — would be + * a cache that never hits and never releases, which is worse than either.) + * + * A sweep interval that is not positive is taken as one millisecond: a timer cannot be finer + * than that, and an interval of zero used to switch the sweeper off, which left the rows to be + * released only when somebody asked for them. + * * @param {number} budget + * @param {{ ttlMs?: number, now?: () => number, sweepIntervalMs?: number }} [options] */ -export function createParsedRowsCache(budget) { - /** @type {Map} */ +export function createParsedRowsCache(budget, options = {}) { + const ttlMs = options.ttlMs ?? PARSED_ROWS_CACHE_TTL_MS; + const now = options.now ?? (() => Date.now()); + // The interval the sweep actually uses, taken between two bounds. + // + // The limit bound is the rule the promise to a reader is made of: the interval is at most a + // quarter of the limit the cache was given, so the rows are gone no later than a quarter of + // the limit after they expire. The clamp is the rule and not a convenience: without it the + // promise would hold for the shipped limit and for nothing else, and the limit is an option a + // caller may set. Its floor is one millisecond, because a timer cannot be finer than that — + // so for a limit below four milliseconds the release is at most a millisecond late rather + // than a quarter of a tiny limit late. + // + // The asked bound is that same floor applied to what the caller asked for. A caller who asks + // for an interval that is not positive used to switch the sweeper off, and a cache whose + // sweeper is off only releases the rows when somebody asks for them — the one moment they + // are wanted again. The floor keeps the sweeper running. + const limitBoundMs = Math.max(1, Math.floor(ttlMs / 4)); + const askedBoundMs = Math.max(1, options.sweepIntervalMs ?? PARSED_ROWS_CACHE_SWEEP_INTERVAL_MS); + const sweepIntervalMs = Math.min(limitBoundMs, askedBoundMs); + /** @type {Map} */ const entries = new Map(); let bytes = 0; + /** @type {ReturnType | null} */ + let sweeper = null; + + /** @param {number} at */ + const expired = (at) => ttlMs > 0 && now() - at >= ttlMs; + + /** @param {string} key */ + const drop = (key) => { + const victim = entries.get(key); + if (victim === undefined) return 0; + entries.delete(key); + bytes -= victim.size; + return victim.size; + }; + + const stopSweeper = () => { + if (sweeper === null) return; + clearInterval(sweeper); + sweeper = null; + }; + + /** + * Release what nobody has wanted for a while. This is what makes the rows go away while the + * reader is gone: a sweep that only ran on a request would release them exactly when the + * reader comes back, which is the one moment they are wanted again. + * + * @returns {number} the bytes still held afterwards + */ + const sweep = () => { + for (const [key, entry] of entries) { + if (expired(entry.at)) drop(key); + } + if (entries.size === 0) stopSweeper(); + return bytes; + }; + + const startSweeper = () => { + // The interval is floored at a millisecond where it is taken, so the third condition cannot + // fire today. It stays because it is the difference between the two ways a non-positive + // interval could hurt: a timer that runs as fast as the platform allows, or no timer at all + // — and no timer means the rows are released only when somebody asks for them. + if (sweeper !== null || ttlMs <= 0 || sweepIntervalMs <= 0) return; + sweeper = setInterval(sweep, sweepIntervalMs); + // An interval that is not unref'd keeps the event loop alive after the server closes, and the + // runtime would wait for it. It must never be the reason the process is still running. + if (typeof sweeper.unref === 'function') sweeper.unref(); + }; + return { get bytes() { return bytes; }, get size() { return entries.size; }, + // The interval the sweep actually uses, so the relation between the limit and the release + // can be checked on the running cache rather than on the two numbers in the source. + get sweepIntervalMs() { return sweepIntervalMs; }, /** * @param {string} key * @returns {ParsedRows | null} @@ -2656,9 +3751,16 @@ export function createParsedRowsCache(budget) { get(key) { const hit = entries.get(key); if (hit === undefined) return null; + if (expired(hit.at)) { + // The rows have been here longer than anybody asked for them. Releasing them here is the + // same release the sweeper would do; the only difference is that this request asked. + drop(key); + return null; + } // Re-reading is the only signal that a file is still hot, so it refreshes recency. entries.delete(key); entries.set(key, hit); + hit.at = now(); return hit.value; }, /** @@ -2674,7 +3776,7 @@ export function createParsedRowsCache(budget) { entries.delete(key); } if (size > budget) return bytes; - entries.set(key, { value, size }); + entries.set(key, { value, size, at: now() }); bytes += size; while (bytes > budget && entries.size > 1) { const oldest = entries.keys().next(); @@ -2683,11 +3785,15 @@ export function createParsedRowsCache(budget) { entries.delete(oldest.value); bytes -= victim ? victim.size : 0; } + startSweeper(); return bytes; }, + /** Release what expired. The timer calls this; a caller may call it too. */ + sweep, clear() { entries.clear(); bytes = 0; + stopSweeper(); }, }; } @@ -2699,7 +3805,7 @@ export function createParsedRowsCache(budget) { * a session that is still being written to mints a NEW key every time it grows — and every miss * stores another complete body. A plain FIFO therefore keeps four versions of the SAME question * alive, three of which no request can reach any more: `/raw` resolves the current key, and the - * page's next poll produces a newer one. Measured on a 9.1 MB session polled every 8s, the cache + * page's next poll produces a newer one. Measured on a 9.1 MB session polled on every tick, the cache * held four bodies (49.6 MB) after four polls; at this machine's 68 MB body that is three dead * copies. * @@ -2713,7 +3819,7 @@ export function createParsedRowsCache(budget) { * @param {number} limit */ export function createTrajectoryCache(limit) { - /** @type {Map} */ + /** @type {Map} */ const entries = new Map(); return { get size() { return entries.size; }, @@ -2729,16 +3835,54 @@ export function createTrajectoryCache(limit) { entries.set(key, hit); return hit.value; }, + /** + * The entry that answers for this family AND for this session, if there is one. + * + * The key names a VERSION; the family names the QUESTION. Two requests that name the same + * directory and the same slice ask the same question, and the answer to it is the entry this + * cache holds -- but the key it was stored under can name a different version of it, because + * the key also carries the requester's context. A poll that asked for `latest` and a read that + * asked for that session by its id are the same reader of the same body, and they have to + * reach the same entry: the first stores the body under a key that carries the binding it + * resolved, the second looks it up under a key that does not, and the two keys differ for + * reasons that have nothing to do with the body. + * + * A family holds exactly one entry -- storing a newer one drops its predecessor -- so what is + * found here is the current answer to that question. + * + * The session is asked for as well, and that is not a second question: the family names the + * DIRECTORY and the slice, and a directory can stop holding one session and start holding + * another. The manifest beside the transcript is the authority on that, it can change under a + * cached walk, and the walk's own decision is not the last word. An entry built for the old + * session is therefore not an answer about the new one, however well the directory and the + * slice match -- and its decoded messages are the old session's messages. + * + * @param {string} family + * @param {string} identityId the session the entry has to be about + * @returns {any | null} + */ + getByFamily(family, identityId) { + for (const [key, hit] of entries) { + if (hit.family !== family) continue; + if (hit.identityId !== identityId) continue; + // Re-reading is the only signal that an entry is still hot, so it refreshes recency. + entries.delete(key); + entries.set(key, hit); + return hit.value; + } + return null; + }, /** * @param {string} key * @param {string} family the (session, slice) this answers for; a newer version of the same * family replaces the older one instead of stacking beside it + * @param {string} identityId the session this body is about * @param {any} value */ - set(key, family, value) { + set(key, family, identityId, value) { const previous = entries.get(key); if (previous !== undefined) entries.delete(key); - entries.set(key, { at: Date.now(), family, value }); + entries.set(key, { at: Date.now(), family, identityId, value }); for (const [existingKey, existing] of entries) { if (existingKey !== key && existing.family === family) entries.delete(existingKey); } @@ -2816,13 +3960,155 @@ export async function loadParsedRows(cache, filePath, limit, options) { return value; } +/** + * One piece of a JSON array, serialized on its own. + * + * The array is cut at element boundaries, and the pieces concatenate back into exactly the + * string `JSON.stringify` would have produced for the whole array — the opening bracket opens + * the first piece, the closing bracket closes the last one, and every element after the first + * carries its comma with it, so a piece boundary never lands between an element and its comma. + * + * An element that does not fit the budget on its own becomes a piece of its own. That is the one + * thing the budget cannot bound, and it is bounded by the element instead of by the budget. + * + * THE CLOSING BRACKET IS COUNTED. It is one byte, it lands on the piece that holds the last + * element, and a piece that held two elements and ended one byte over the budget was a piece + * the budget did not bound. One byte is reserved for it on every piece, so no piece that holds + * more than one element is ever bigger than the budget. + * + * @param {Array} items + * @param {number} budgetBytes + * @returns {Array} + */ +function jsonArrayPieces(items, budgetBytes) { + const pieces = []; + let current = '['; + let currentBytes = 1; + let first = true; + for (const item of items) { + const encoded = JSON.stringify(item); + // JSON.stringify writes an undefined element as null, and so does this. + const text = encoded === undefined ? 'null' : encoded; + const textBytes = Buffer.byteLength(text, 'utf8'); + // The comma is counted but not concatenated into the piece: `',' + text` would be a + // cons string, and measuring its length would flatten it into a second copy of the + // element. On a body of fifty thousand records that is fifty thousand copies. + const pieceBytes = textBytes + (first ? 0 : 1); + // The closing bracket is counted here, not appended at the end. It lands on the piece that + // holds the LAST element, and a piece that ended exactly at the budget came out one byte + // over it — so the budget did not bound the piece it was supposed to bound. One byte is + // reserved on every piece, which is one byte more than the pieces that do not carry the + // bracket need: the budget is a ceiling, not a target. + if (currentBytes > 1 && currentBytes + pieceBytes + 1 > budgetBytes) { + pieces.push(Buffer.from(current, 'utf8')); + current = ''; + currentBytes = 0; + } + if (!first) { + current += ','; + currentBytes += 1; + } + current += text; + currentBytes += textBytes; + first = false; + } + current += ']'; + pieces.push(Buffer.from(current, 'utf8')); + return pieces; +} + +/** + * The trajectory body, serialized in pieces small enough that no single string ever holds it. + * + * THE BYTES ARE THE SAME BYTES. Concatenating the pieces gives exactly what + * `JSON.stringify(payload)` would have produced — same key order, same escaping, same numbers — + * so the page, the search, `/raw` and `?since=` cannot tell the difference. A guard pins that + * byte for byte, because a body that is *nearly* the same is a body the page cannot parse. + * + * WHAT CHANGES IS HOW THEY ARE PRODUCED. The payload's big array is cut into pieces at element + * boundaries, each piece is serialized on its own and turned into bytes immediately, and the + * string is let go before the next piece is built. So the process never holds THAT ARRAY as one + * string, and never holds the builder's intermediate copies either. + * + * THE EXCEPTION IS A VALUE THAT IS NOT AN ARRAY. It is serialized on its own, by + * `JSON.stringify(value)`, and appended to the piece being built — so a payload whose bulk is one + * large string still holds that string, and the piece that holds it is a piece like any other. + * What this change removes is the array. + * + * WHY IT IS WORTH THE ROUTE'S ATTENTION. Measured on 2026-10-23 on this machine, on the code as + * it stood before the body was chunked, on one session of 57,436 records: one cold read left the + * process at 1,199 MB resident with a 302.7 MB live heap, and the peak inside the route was + * 1,331 MB resident on 1,115 MB of live heap — the payload object graph was 366 MB, and the single + * `JSON.stringify` that followed it added 749 MB on top. Clearing every cache afterwards freed + * the live heap down to 8.4 MB and the process still held 1,070 MB: the pages this request + * touched are not given back. + * + * And the floor does NOT follow the peak. Chunking the body took 151 MB off the high-water mark + * and the floor did not move (1,070.1 MB before, 1,069.7 MB after). So the floor is neither what + * the cache holds nor what the peak left behind: it is what this request allocated, and cutting + * the allocation is the lever. + * + * @param {Record} payload + * @param {number} [budgetBytes] + * @returns {Array} + */ +export function serializeTrajectoryBody(payload, budgetBytes = TRAJECTORY_BODY_CHUNK_BYTES) { + const pieces = []; + let current = ''; + let currentBytes = 0; + let needsComma = false; + + const flush = () => { + if (currentBytes === 0) return; + pieces.push(Buffer.from(current, 'utf8')); + current = ''; + currentBytes = 0; + }; + const append = (text) => { + current += text; + currentBytes += Buffer.byteLength(text, 'utf8'); + }; + + append('{'); + for (const key of Object.keys(payload)) { + const value = payload[key]; + // What JSON.stringify leaves out, this leaves out: a key whose value is undefined, a + // function or a symbol is not in the JSON at all. + if (value === undefined || typeof value === 'function' || typeof value === 'symbol') continue; + if (Array.isArray(value)) { + if (needsComma) append(','); + append(JSON.stringify(key) + ':'); + // The array goes out in pieces, so the string it would have been never exists. + flush(); + for (const piece of jsonArrayPieces(value, budgetBytes)) pieces.push(piece); + needsComma = true; + continue; + } + const encoded = JSON.stringify(value); + // A value whose toJSON() returns undefined is left out by JSON.stringify as well. + if (encoded === undefined) continue; + if (needsComma) append(','); + append(JSON.stringify(key) + ':' + encoded); + needsComma = true; + } + append('}'); + flush(); + return pieces; +} + /** * Sends an already-serialized JSON body. Unlike {@link sendJson} it supports revalidation: * `no-store` would suppress the 304 path, so revalidated responses advertise `no-cache` * (storable, but must be revalidated) and carry the tag the page echoes back. + * + * The body is either one string or a list of byte pieces. The list is what the trajectory route + * sends now: the bytes are the same bytes, but the payload's big array was never held as one + * string, so the declared length is the sum of the pieces rather than the length of a string. + * (A value inside the payload that is not an array is still one string — see + * `serializeTrajectoryBody`.) * @param {any} response * @param {number} status - * @param {string} body + * @param {string | Buffer | Array} body * @param {{ etag?: string, revalidate?: boolean }} [options] */ export function sendJsonBody(response, status, body, options) { @@ -2840,10 +4126,20 @@ export function sendJsonBody(response, status, body, options) { } return; } - headers['content-length'] = Buffer.byteLength(body, 'utf8'); + const pieces = Array.isArray(body) ? body : [body]; + let length = 0; + for (const piece of pieces) { + length += Buffer.isBuffer(piece) ? piece.length : Buffer.byteLength(piece, 'utf8'); + } + headers['content-length'] = length; try { response.writeHead(status, headers); - response.end(body); + if (pieces.length === 1 && typeof pieces[0] === 'string') { + response.end(pieces[0]); + return; + } + for (const piece of pieces) response.write(piece); + response.end(); } catch { // The client polls aggressively and aborts requests; a closed socket is not an error here. } @@ -2859,10 +4155,33 @@ function sendError(response, status, code, message) { sendJson(response, status, { error: code, message }); } +/** + * The plugin package installs onto every reader's machine, so nothing derived may be written inside + * it. `context.dataDir` is the runtime's own data directory; a caller that points it at the + * package, or at anything under it, has the search segments land in a directory that ships to + * readers. + * + * The refusal names the path it refused, because the caller is the one that has to change it, and a + * refusal that does not say which path was refused is a refusal nobody can act on. + * + * @param {MiniAppContext} context + */ +function assertDataDirOutsidePlugin(context) { + const dataDir = typeof context.dataDir === 'string' ? context.dataDir : ''; + if (!dataDir) throw new Error('the runtime did not give this plugin a data dir to write into'); + const pluginRoot = typeof context.pluginRoot === 'string' ? context.pluginRoot : ''; + if (isUnderDirectory(pluginRoot, dataDir)) { + throw new Error( + `the data dir must not be inside the plugin package: ${resolve(dataDir)} is under ${resolve(pluginRoot)}` + ); + } +} + /** * @param {MiniAppContext} context */ export async function start(context) { + assertDataDirOutsidePlugin(context); const location = resolveSessionsRoot(); const secrets = [location.sessionsRoot, location.root, join(homedir(), '.minimax')]; const redact = createRedactor(secrets); @@ -2944,9 +4263,9 @@ export async function start(context) { * * readSessionTitles opens the runtime database on every call, and the list route calls it once * per poll for rows that only change when somebody renames a session. Raising the walk's TTL - * without this would still reopen a 700 MB store every 2.5 s forever, so the two caches answer + * without this would still reopen a 700 MB store on every poll forever, so the two caches answer * the same poll or neither does. - * @type {{ at: number, key: string, map: Map }} + * @type {{ at: number, key: string, map: Map }} */ let titleCache = { at: 0, key: '', map: new Map() }; /** @@ -2968,11 +4287,14 @@ export async function start(context) { }; const describeSession = async (entry) => { - const cacheKey = `${entry.dir}|${entry.sizeBytes}|${entry.lastActiveAt}`; + // The id is the walk's decision, and the walk read the manifest to make it, so the row says + // the same thing about the session whether it is read here or by the layer or by the + // validator. The id is in the cache key: a manifest that changes changes the id, and a + // descriptor cached under the old one would publish the old session's id. + const cacheKey = `${entry.dir}|${entry.sizeBytes}|${entry.lastActiveAt}|${entry.sessionId}`; const cached = descriptorCache.get(cacheKey); if (cached && Date.now() - cached.at < SESSION_CACHE_TTL_MS) return cached.value; const identity = await readSessionIdentity(entry.dir); - const cacheId = `${entry.dir}|${identity.id}`; let peek = { messages: 0, userMessages: 0, assistantMessages: 0, toolResults: 0, toolCalls: 0, thinkingBlocks: 0, turns: 0 }; let label = null; let startedAt = identity.createdAtMs; @@ -2991,8 +4313,8 @@ export async function start(context) { context.logger.warn('miniapp.trajectory.read_error', { reason: 'peek_failed', code: errorCode(error) }); } const value = { - id: identity.id, - label: label ?? `会话 ${String(identity.id).slice(4, 12)}`, + id: entry.sessionId, + label: label ?? `会话 ${String(entry.sessionId).slice(4, 12)}`, // A prefix count is a prefix count. Reported as the session's message count it is simply // untrue, so when the prefix was not the whole file these are absent rather than approximate — // and the page's job is to print nothing rather than to print a number it cannot stand behind. @@ -3003,7 +4325,7 @@ export async function start(context) { // here, and /api/trajectory's is. Same name, different scope. Pinned by tests/verify-list-cache.mjs. sizeBytes: entry.sizeBytes, turnCount: peekIsWhole ? peek.turns : null, - cacheId, + cacheId: `${entry.dir}|${entry.sessionId}`, }; descriptorCache.set(cacheKey, { at: Date.now(), value }); if (descriptorCache.size > SESSION_CACHE_ENTRIES) { @@ -3085,31 +4407,75 @@ export async function start(context) { const handleSessions = async (response, url) => { const limit = normalizeLimit(url.searchParams.get('limit'), 20, 1, 50); - let entries; + let candidates; try { - // The picker still sees only the most recent sessions. The id index beside it is what - // `handleTrajectory` resolves explicit ids against, and it is deliberately not what the - // picker is built from — widening the list is a separate decision about the UI. - entries = (await listEntries()).recentEntries; + // The picker is built from the walk's own list: the most recent sessions, mtime-descending. + // That list is also what 'latest' falls back to when the runtime database is unavailable, + // and nothing here reorders it. The VISIBLE set below is a different set — it draws from + // the MAIN sessions only, and the rule for picking twenty of them is no longer pure mtime: + // the current conversation comes first, then the sessions of its workspace. The capped + // window is deliberately NOT the pool: a sub-agent sits in that window and would take a + // main session' place, and the walk has already stat'ed every candidate besides, so the + // deeper pool costs no extra disk work. + candidates = await listEntries(); } catch (error) { context.logger.error('miniapp.trajectory.read_error', { reason: 'root_unreadable', code: errorCode(error) }); sendError(response, 503, 'trajectory_unavailable', '未找到本地会话数据目录'); return; } - if (entries.length === 0) { + const allEntries = candidates.allEntries; + if (allEntries.length === 0) { sendError(response, 503, 'trajectory_unavailable', '未找到本地会话数据目录'); return; } - const visible = entries.slice(0, limit); - // Real titles from the runtime database; sub-agent and cron sessions get a kind badge so - // the picker never presents them as if they were the user's conversation. - const titles = await sessionTitles( - visible.map((entry) => decodeSessionIdFromDirName(basename(entry.dir))).filter(Boolean), - ); - // Built from the same two values the payload below is built from, and computed before any - // descriptor work — which is what lets an unchanged list answer without building it. - const etag = weakETag(sessionsValidator(visible, titles, limit)); + // The conversation the runtime database reports as current, asked of the database rather than + // guessed from mtime: the conversation being read is not necessarily the one whose file was + // written last. What the database reports is a ranking, not a certainty -- see + // `resolveCurrentConversation`, which says so where it happens. + const current = await resolveCurrentConversation(location.root); + + // Real titles from the runtime database, for every candidate and for the current conversation, + // which may be outside the walk. The workspace column rides along because the visible set is + // ordered by it, and the parent column rides along because the layer is decided by it. + // + // The ids are the ones the walk published those directories under -- not the ids their names + // happen to decode to. A directory whose name does not decode is identified by its manifest, + // and the runtime database knows it under that id. + const ids = allEntries.map((entry) => entry.sessionId).filter(Boolean); + if (current && current.sessionId && ids.indexOf(current.sessionId) < 0) ids.push(current.sessionId); + const titles = await sessionTitles(ids); + + // The two layers. A sub-agent is a session like any other on disk, and the runtime hangs it + // off its parent with one column; so the layer is a fact about a row, and the page gets the + // main sessions in the picker and the sub-agents of whichever main is in hand below it. + const layers = splitSessionLayers(allEntries, titles); + + // The visible set: the current conversation first, then the sessions of its workspace, then + // the rest -- each group mtime-descending. At most one session joins the window this way, and + // it is always the current conversation. The number of visible sessions is unchanged. + const visible = await buildVisibleSessionList(layers.mains, current, titles, location.sessionsRoot, limit, () => { + context.logger.warn('miniapp.trajectory.read_error', { reason: 'session_dir_rejected' }); + }); + + // Every run in the pool, and not only the runs of the mains that are on the page. + // + // The picker is two controls now: the conversation, and the runs inside it. The second one + // lists the runs of the conversation the reader picked -- which may be a conversation that + // has aged out of the twenty, the one the page keeps alive with a row of its own. Dropping + // those runs would break the promise the main list already keeps: the session you are + // reading does not disappear because it stopped being new. The order inside each group is + // the one `splitSessionLayers` set -- most recently active first. + const childEntries = []; + for (const bucket of layers.childrenByParent.values()) { + for (const child of bucket) childEntries.push(child); + } + + // Built from the same values the payload below is built from, and computed before any + // descriptor work -- which is what lets an unchanged list answer without building it. The + // sub-agents are in it: a new one changes what the page shows, and a validator that did not + // move would let the caller keep a body that never had it. + const etag = weakETag(sessionsValidator(visible, titles, limit, childEntries)); const requestEtag = response.req && response.req.headers ? response.req.headers['if-none-match'] : null; if (etagMatches(requestEtag, etag)) { sendJsonBody(response, 304, '', { etag, revalidate: true }); @@ -3121,22 +4487,57 @@ export async function start(context) { // to produce. let body = sessionsBody.material === etag ? sessionsBody.body : null; if (body === null) { + // One writer for a row, so the main sessions and the sub-agents cannot drift apart in the + // fields they carry: the same title, the same kind, the same tag, the same floor. + const describeRow = async (entry) => { + const descriptor = await describeSession(entry); + // The runtime facts are looked up under the id the row is published under, so the row's + // id and the row's facts are the same session by construction rather than by agreement. + const meta = titles.get(entry.sessionId); + // The row is BUILT, not patched. The descriptor comes out of a cache and is shared, so + // writing the runtime's facts onto it leaves the last writer's facts on it for the next + // reader -- and a session whose row has left the runtime database goes on publishing the + // title, the kind and the parent it had last time, while the validator, which reads the + // same rows, reports none of them. A body and an ETag that disagree about the same row is + // how a badge outlives the fact it was derived from. + // + // So every runtime fact is written on every pass, and a row with no runtime facts carries + // none of them: the label falls back to the one the filesystem gives, and the kind, the + // parent and the tag are null. + return { + ...descriptor, + label: meta && meta.title ? meta.title : descriptor.label, + sessionKind: meta ? meta.kind : null, + sessionParentId: meta ? meta.parent : null, + // What this session IS, as a token the page can render: a lightweight exchange, a + // scheduled run, a background task. The raw purpose stays on this side of the wire -- + // it is free text, and free text is not something to hand a reader's browser. + sessionTag: meta ? sessionTagFor(meta.kind, meta.purpose) : null, + }; + }; const described = []; for (const entry of visible) { try { - const descriptor = await describeSession(entry); - const meta = titles.get(decodeSessionIdFromDirName(basename(entry.dir))); - if (meta) { - if (meta.title) descriptor.label = meta.title; - descriptor.sessionKind = meta.kind; - descriptor.sessionParentId = meta.parent; - } - described.push(descriptor); + described.push(await describeRow(entry)); + } catch (error) { + context.logger.warn('miniapp.trajectory.read_error', { reason: 'describe_failed', code: errorCode(error) }); + } + } + // The sub-agents, as a flat list: each row names the main it hangs off, and the page groups + // them. A flat list is what lets the two layers join on ONE id -- the id the main row + // already carries -- without a second naming scheme to keep in step with it. + const children = []; + for (const entry of childEntries) { + try { + children.push(await describeRow(entry)); } catch (error) { context.logger.warn('miniapp.trajectory.read_error', { reason: 'describe_failed', code: errorCode(error) }); } } - body = JSON.stringify({ sessions: described.map(stripInternalFields) }); + body = JSON.stringify({ + sessions: described.map(stripInternalFields), + children: children.map(stripInternalFields), + }); sessionsBody = { material: etag, body }; } sendJsonBody(response, 200, body, { etag, revalidate: true }); @@ -3158,7 +4559,7 @@ export async function start(context) { return { ok: false, status: 503, code: 'trajectory_unavailable', message: '未找到本地会话数据目录' }; } const entries = candidates.recentEntries; - if (entries.length === 0 && candidates.byDecodedId.size === 0) { + if (entries.length === 0 && candidates.bySessionId.size === 0) { return { ok: false, status: 503, code: 'trajectory_unavailable', message: '未找到本地会话数据目录' }; } const requested = url.searchParams.get('session'); @@ -3169,41 +4570,54 @@ export async function start(context) { // a worker or cron session often has the newest file even while the user reads another one. const current = await resolveCurrentConversation(location.root); if (current) { - entry = entries.find((candidate) => basename(candidate.dir).includes('session_') - && decodeSessionIdFromDirName(basename(candidate.dir)) === current.sessionId) ?? null; + // The current session's own row, found by the id its directory name decodes to and + // CONFIRMED by the manifest beside the transcript: the name is a claim, the manifest is + // the authority. A directory whose name says this session and whose manifest says another + // one is that other session's directory, and binding this session's id to it would serve + // somebody else's messages under this session's name. + entry = await verifiedEntryForSessionId(entries, current.sessionId); if (!entry && current.relativeDir) { // Not in the recent window — build the entry straight from the recorded path. That value // is a database column rather than a name this app chose, so it earns the same - // containment check `resolveArtifactPath` gives catalog file names. - const root = resolve(location.sessionsRoot); - const dir = resolve(root, current.relativeDir.replace(/\\/g, '/')); - if (dir !== root && !dir.startsWith(root + sep)) { + // containment check `resolveArtifactPath` gives catalog file names. The picker builds the + // same kind of entry from the same column, so both go through one function. + // + // The session it is supposed to be is named too: a path that points at a different + // session's directory is refused, and 'latest' falls back to the walk's answer rather + // than binding this session's id to somebody else's transcript. + const built = await entryFromRelativeDir(location.sessionsRoot, current.relativeDir, current.sessionId); + if (built.rejected) { context.logger.warn('miniapp.trajectory.read_error', { reason: 'session_dir_rejected' }); - entry = null; - } else { - try { - const fileStat = await stat(join(dir, 'messages.jsonl')); - entry = { dir, sizeBytes: fileStat.size, lastActiveAt: Math.floor(fileStat.mtimeMs) }; - } catch { - entry = null; - } } + entry = built.entry; } if (entry) binding = current; } - if (!entry) entry = entries[0]; + if (!entry) { + // The window's first entry -- but only while it still holds the session it was + // published under. The explicit-id routes ask that again before they serve a + // candidate, and the manifest can change under the walk, so the fallback has to ask + // it too: a directory whose manifest no longer agrees with its name is nobody's + // directory, and serving it here while both explicit ids refuse it would be the + // same transcript under an id the reader cannot ask for. + entry = await firstHoldingEntry(entries); + if (!entry) { + return { ok: false, status: 404, code: 'session_not_found', message: '找不到可用的会话' }; + } + } } else { // Two ways in, and the order matters. // - // Fast path: the id was decoded from a directory name during the walk, so the directory is + // Fast path: the id is the one the walk published that directory under, so the directory is // already known and no scanning is needed. This is what makes a session outside the recent - // 30 reachable at all — before this, the only directories this branch could see were the 30 - // the picker shows, so 116 of 146 sessions on this machine answered 404 to their own id. + // window reachable at all — before this, the only directories this branch could see were the ones + // the picker shows, so a session outside that window answered 404 to its own id. // - // The decoded name is still only a claim. `readSessionIdentity` prefers `manifest.sessionId`, - // so a directory whose name disagrees with its manifest must not be served under the - // requested id; it falls through to the slow path instead, which re-checks every candidate. - const fastPath = candidates.byDecodedId.get(requested); + // The walk's decision is not the last word. The manifest can change under the walk, so the + // identity of the directory is read again and compared: a directory that has stopped + // holding this session is not served under it, and the request falls through to the slow + // path, which re-checks every candidate. + const fastPath = candidates.bySessionId.get(requested); let picked = null; if (fastPath) { // Duplicates are kept as a list rather than collapsed, because collapsing them with @@ -3225,7 +4639,14 @@ export async function start(context) { const matched = []; for (const candidate of entries) { const candidateIdentity = await readSessionIdentity(candidate.dir); - if (candidateIdentity.id === requested || basename(candidate.dir) === requested) { + // The same rule the walk applies, asked again of the directory that is in hand: a + // directory whose name decodes to one session and whose manifest names another is + // nobody's directory, and it is not served under either id. The walk decided this + // before the manifest changed, so the manifest is read again here and the answer + // follows the manifest that is on disk now. + const claimed = decodeSessionIdFromDirName(basename(candidate.dir)); + const holds = !claimed || claimed === candidateIdentity.id; + if (holds && (candidateIdentity.id === requested || basename(candidate.dir) === requested)) { matched.push({ entry: candidate, identity: candidateIdentity }); break; } @@ -3268,7 +4689,23 @@ export async function start(context) { // Everything needed to decide "did this change?" is already in hand. Building the key // before the read is what lets an unchanged session skip the read entirely. - return { ok: true, entry, binding, identity, generations, filter, cacheKey: trajectoryCacheKey(entry, identity, binding, filter) }; + // + // The family travels with it: the key names the version and the family names the question, + // and a read that asks for this session by its id has to be able to reach the entry a poll + // stored for it even though the two computed different keys. + return { + ok: true, + entry, + binding, + identity, + // The session this request is about, for the reader that has to tell one session's cached + // body from another's. + identityId: identity.id ?? '', + generations, + filter, + cacheKey: trajectoryCacheKey(entry, identity, binding, filter), + family: trajectoryCacheFamily(entry.dir, filter), + }; }; /** @@ -3500,7 +4937,12 @@ export async function start(context) { truncated, redact, generations, - generation: shown.length === 1 ? shown[0].generation : null, + // The slice this body IS, which is the slice that was asked for. It used to be "the one + // generation the body happens to show", and that is a different question: a body built for + // the whole lineage and a body built for one generation have to be told apart, because the + // page compares a search answer against the body on screen before it lets that answer's + // indexes filter it. + generation: filter, orphans: lineage.orphans, identity: identityTable, openerByTurn: view.openerByTurn, @@ -3539,6 +4981,11 @@ export async function start(context) { const locator = rowLocators[rowIndex]; if (locator !== undefined) raws.set(recordIndex, locator); } + // The version of every record, stamped before the body is serialized. A record index is a + // position inside ONE body, and the same position in a newer body can be a different record: + // the page asks about positions all the time (the search answer, `/raw`, `/record`), so the + // position carries the version of the record it means. See `recordVersionToken`. + stampRecordVersions(payload, raws); // Last thing before serializing, and the only place the bulk fields are ever taken off a // record. It happens here rather than at `push()` because the search, which runs in this @@ -3550,7 +4997,11 @@ export async function start(context) { let body; try { - body = JSON.stringify(payload); + // In pieces, not as one string. The bytes are the same bytes — see serializeTrajectoryBody + // for why that is a contract and not an accident — but the process never holds the + // payload's big array as a string, and the peak this route used to leave behind does not + // come back. + body = serializeTrajectoryBody(payload); } catch { sendJson(response, 500, { error: 'trajectory_unserializable', message: '会话数据暂时无法读取' }); return; @@ -3559,7 +5010,7 @@ export async function start(context) { // same while it goes stale. Everything that decides whether the answer changed stays in the // key, so a newer version of one family replaces its predecessor instead of stacking beside it. // `decoded` is filled in by /raw, at most RAW_DECODE_CACHE_ENTRIES lines' worth. - trajectoryCache.set(cacheKey, trajectoryCacheFamily(entry.dir, filter), { at: Date.now(), etag, body, raws, rowPos: rowPosByRecord, decoded: new Map() }); + trajectoryCache.set(cacheKey, trajectoryCacheFamily(entry.dir, filter), identity.id ?? '', { at: Date.now(), etag, body, raws, rowPos: rowPosByRecord, decoded: new Map() }); // Nothing below reads `rows` or `rowLocators`, and `payload` holds its own projections rather // than the rows, so this is where the object graph stops being reachable. A sealed generation's // rows are still held by the parsed-rows cache — that is the point of that cache — but the @@ -3571,6 +5022,23 @@ export async function start(context) { // and not the server's work. Any reading of "41 MB → a few KB, so polling got faster" // has to carry that with it. Pinned by tests/verify-incremental.mjs. const since = parseSince(url.searchParams.get('since')); + if (since !== null) { + // WHICH SET the tail is about. The tail carries the records after the cursor; the ones + // before it stay as the page has them, so the tail is only about the page's copy if that + // copy is still this body's. A record rewritten in place keeps its index and its id, so + // the cursor still points at the same record and the tail looks empty — the page would + // keep the old text under the new body's validator, and nothing else in the protocol + // could see that it had changed. + const refusal = deltaRefusal( + payload, + Number.parseInt(url.searchParams.get('n') ?? '', 10), + url.searchParams.get('fp') ?? '' + ); + if (refusal) { + sendError(response, refusal.status, refusal.code, refusal.message); + return; + } + } const delta = since === null ? null : buildDeltaPayload(payload, since); sendJsonBody(response, 200, delta === null ? body : JSON.stringify(delta), { etag, revalidate: true }); }; @@ -3613,6 +5081,25 @@ export async function start(context) { const needle = foldNeedle(asked); const { entry, filter } = ctx; + // WHICH BODY THE ANSWER IS ABOUT. The page says which body that is: the records it has, in + // order, as a fingerprint. The service folds its own records the same way and compares, and + // refuses to answer when the two are not the same set. + // + // Without this the service can only answer about the body on disk, and the page then filters + // its own body with an answer about another one. What hurts is the part the answer does NOT + // name: a record this body has and that body does not match gets dimmed as if it did not + // contain the needle, with no error anywhere, and the reader is told the record is not there. + // + // The count matters as much as the fingerprint. The fold is over the FIRST N records, because + // the page is behind whenever the session grew since its last poll — and a session that grew + // has MORE records, not different ones, so the first N are still the same N. + const count = Number.parseInt(url.searchParams.get('n') ?? '', 10); + const askedFingerprint = (url.searchParams.get('fp') ?? '').trim(); + if (!Number.isInteger(count) || count < 0 || !/^[0-9a-f]{16}$/.test(askedFingerprint)) { + sendError(response, 400, 'search_fingerprint_required', '会话已更新，请重新加载后再搜索'); + return; + } + let walked; try { walked = await collectLineage(entry, ctx.generations); @@ -3624,20 +5111,38 @@ export async function start(context) { const sessionKey = searchSessionKey(entry.dir, ctx.identity.id); const viewKey = searchViewKey(filter); - // Before anything expensive. The lineage walk above is stat-driven and cheap; everything after - // this line is the rebuild, and on the largest session measured here it was 3.7 seconds for a - // search that returns at most 500 rows. - const onDisk = await searchFromArtifacts({ sessionKey, viewKey, files, needle }); - if (onDisk !== null) { + // ONE answer, sent by whichever path found the hits. + // + // Two paths answering the same request with two different response shapes is not a faster + // version of the endpoint, it is a second endpoint — and the second one is where the slice + // the answer is about would have gone stale. The slice travels with every answer, in the + // same word the body uses, so the page can compare the two. + // + // @param {Array} found the hits + // @param {number} foundTotal every match, which is not every hit + const sendAnswer = (found, foundTotal) => { sendJson(response, 200, { query: asked, - hits: onDisk.hits, - total: onDisk.total, - truncated: onDisk.total > onDisk.hits.length, - // The same field the rebuild sends. Two paths answering the same request with two different - // response shapes is not a faster version of the endpoint, it is a second endpoint. + hits: found, + total: foundTotal, + truncated: foundTotal > found.length, session: { id: ctx.identity.id, generation: filter }, }); + }; + + // Before anything expensive. The lineage walk above is stat-driven and cheap; everything after + // this line is the rebuild, and on the largest session measured here it was 3.7 seconds for a + // search that returns at most MAX_SEARCH_HITS rows. + const onDisk = await searchFromArtifacts({ sessionKey, viewKey, files, needle, count }); + if (onDisk !== null) { + // The segments ARE the body this answer is about, so their records decide whether it is + // about the body the page is showing. The fold is over the first N, for the reason given + // above: a page that is behind has fewer records, not different ones. + if (onDisk.fingerprint !== askedFingerprint) { + sendError(response, 409, 'search_version_mismatch', '该记录所在的版本已变化，请重新加载会话'); + return; + } + sendAnswer(onDisk.hits, onDisk.total); return; } @@ -3657,8 +5162,25 @@ export async function start(context) { }); const rowLocators = view.occurrences.map((occurrence) => occurrence.locator); const rowIndexByRecord = splitRawRecords(payload.turns); + // The record -> its line's locator, so every record can be stamped with its version before it + // is serialized or written into a segment. The search answer and the body the page filters with + // have to carry the SAME version for the same record, or the page is comparing two answers. + /** @type {Map} */ + const locatorsByRecord = new Map(); + for (const [recordIndex, rowIndex] of rowIndexByRecord) { + const locator = rowLocators[rowIndex]; + if (locator !== undefined) locatorsByRecord.set(recordIndex, locator); + } + stampRecordVersions(payload, locatorsByRecord); - /** @type {Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null }>} */ + // The same check, on the same records, on the path that rebuilt them. The fold is over the + // first N records of THIS body, which is what the page folded over the first N of its own. + if (recordSetFingerprint(payloadRecordVersions(payload, count)) !== askedFingerprint) { + sendError(response, 409, 'search_version_mismatch', '该记录所在的版本已变化，请重新加载会话'); + return; + } + + /** @type {Array<{ index: number, turnIndex: number, kind: string, summary: string, generation: number | null, v: string }>} */ const hits = []; let total = 0; const searched = collectSearchHits(payload, needle, MAX_SEARCH_HITS); @@ -3678,13 +5200,7 @@ export async function start(context) { activeFileName: walked.activeFileName, }).catch(() => undefined); - sendJson(response, 200, { - query: asked, - hits, - total, - truncated: total > hits.length, - session: { id: ctx.identity.id, generation: filter }, - }); + sendAnswer(hits, total); // Nothing below reads the view, and the payload holds projections rather than rows, so this is // where the object graph stops being reachable. view.rows.length = 0; @@ -3710,18 +5226,30 @@ export async function start(context) { * The session's record index is NOT read. Each segment line carries its own answer, so a needle * matching 14 records out of 36,525 reads those 14 lines rather than parsing 34.2 MB of index. * - * @param {{ sessionKey: string, viewKey: string, files: Array, needle: string }} input + * @param {{ sessionKey: string, viewKey: string, files: Array, needle: string, count: number }} input */ - const searchFromArtifacts = async ({ sessionKey, viewKey, files, needle }) => { + const searchFromArtifacts = async ({ sessionKey, viewKey, files, needle, count }) => { /** @type {Array<{ index: number, turnIndex: number | null, kind: string, summary: string, generation: number | null }>} */ const hits = []; let total = 0; + // The ledger position of every record these segments cover: the turn it is in and its own + // number in the body. They are what the version tokens are folded over IN THE ORDER THE LEDGER + // SHOWS THEM — the segments are one file per generation, and the body is not in file order, + // so the first N records of the files are not the first N records of the body. + /** @type {Array<{ turnIndex: number | null, index: number | null, v: string }>} */ + const positions = []; // `files` is the lineage walk's own output, so a generation that is no longer in the lineage // has no entry here and no segment is read — a leftover on disk cannot keep it searchable. for (let fileIdx = 0; fileIdx < files.length; fileIdx += 1) { const file = files[fileIdx]; const segment = await segmentStore.read(sessionKey, viewKey, generationKey(file.generation), [file]); if (segment === null) return null; + // A set that does not cover the whole body is not the set the page is holding: the records + // no segment covers are missing from the fold, so the fingerprint is the fingerprint of a + // DIFFERENT set of records, and the answer would be refused for a body that did not change. + // The rebuild answers instead — it has the body, and the body is the authority on which + // records exist. + if (segment.manifest.uncovered > 0) return null; const ordinals = scanSegment(segment.surface, needle); const answers = readSegmentHits(segment.surface, ordinals); for (const hit of answers) { @@ -3729,14 +5257,45 @@ export async function start(context) { // does not agree with this reader. Answering without it would quietly drop a record. if (hit === null) return null; total += 1; - if (hits.length < MAX_SEARCH_HITS) hits.push(hit); + // Every match is kept, and the cap is applied later. The cap is about WHICH records the + // answer is about, and the records that come first in the ledger's order are not the + // records that come first in the generations: a record that belongs to an earlier turn and + // arrived after that turn was written sits in a later file, and the ledger still draws it + // first. Truncating here would keep the first N in the order the generations happened to + // be in — a different SET of records, with the same count and the same fingerprint, which + // is the hardest kind of difference for anything downstream to notice. + hits.push(hit); } + // The position and the version of every record this generation holds, read from the surface + // the segment already holds rather than from a second copy of it: both are on every line, + // and a second copy would be a second thing to keep in step. + for (const one of segmentLedgerPositions(segment.surface)) positions.push(one); } - // Ascending ledger order, because the segments were read oldest generation first and a hit's - // place in that list is what the page scrolls to. Sorting by the index itself is the order the - // rebuild produces, and two answers that differ only in order are two answers. - hits.sort((a, b) => a.index - b.index); - return { hits, total }; + // The version tokens of the first N records, in the order the ledger shows them. This is the + // same fact the rebuild folds over the payload's own records, and the two have to agree: the + // fingerprint is what says whether this answer is about the body the page is showing. + const versions = segmentRecordVersions(positions, count); + // A record that cannot be placed in the ledger's order cannot be counted into a set, so the + // answer would be about a set nobody can name. The rebuild answers instead. + if (versions === null) return null; + // The ledger's order, which is the order the rebuild returns its hits in: the turns in the + // order the body numbers them, and inside a turn the records in the order they were + // emitted. The segments were read oldest generation first, so that is NOT the order they + // arrive in, and the record's own number is not it either — a record that belongs to an + // earlier turn and arrived after that turn was written is emitted later than a record of a + // later turn, and the ledger still draws it first. Two answers to one question that differ + // only in order are two answers, so the cached path reports its hits in the same order the + // rebuild does. + hits.sort(byLedgerOrder); + // The cap is applied HERE, after the order is restored — the first N in the ledger's order is + // what the rebuilt path returns, and the two paths have to answer with the same records, not + // merely with the same number of them. + // + // The cost is that every match is held for the length of the walk. That is the same order of + // magnitude as the body this path exists to avoid reading, and it is released with the + // request. Capping the cache by bytes is a separate piece of work. + const capped = hits.length > MAX_SEARCH_HITS ? hits.slice(0, MAX_SEARCH_HITS) : hits; + return { hits: capped, total, fingerprint: recordSetFingerprint(versions) }; }; /** @@ -3757,6 +5316,11 @@ export async function start(context) { if (locator !== undefined) locators.set(recordIndex, locator); } const fileIdxRows = fileIdxRowsFromPayload(payload, locators, files); + // How many of the body's records no segment will hold. A record with no locator names no + // file, so it belongs to no generation's segment — and the segments then cover fewer records + // than the body, which is a different set with a different fingerprint. The count goes into + // every manifest, so a reader can SEE that the set is short instead of answering from it. + const uncovered = uncoveredRecordCount(fileIdxRows); // One segment per generation, the active one included. §8.8.1 rule 5 excluded the active // generation on the grounds that a segment for it would be stale before it was written — true, @@ -3801,13 +5365,16 @@ export async function start(context) { // rebuild forever, for a generation that could never match anything. const generation = files[fileIdx].generation; const generationKeyValue = generationKey(generation); - // Already there and still matching its source: nothing to do, which is the common case on a - // session that is not being compacted right now. const existing = await segmentStore.read(sessionKey, viewKey, generationKeyValue, [files[fileIdx]]); - if (existing !== null) continue; + // Already there and still matching its source: nothing to do, which is the common case on a + // session that is not being compacted right now. The count has to agree as well: it is + // written into every manifest, and a set whose manifests say different things about what + // they cover is not a set a reader can take on faith. + if (existing !== null && existing.manifest.uncovered === uncovered) continue; const segment = buildSegment({ records }); await segmentStore.write(sessionKey, viewKey, generationKeyValue, segment, { generation, + uncovered, source: [files[fileIdx]], }); } @@ -3815,21 +5382,42 @@ export async function start(context) { }; /** - * One record's untouched message, fetched only when something actually reads it. + * The refusal a position read gets when it does not name the record it means. * - * The body no longer has the parsed rows behind it, so the line is read back out of the file it - * came from. That is a seek and a digest over one line, against the alternative of holding every - * session's object graph in memory for the life of the page — measured at 233 MB of the 378 MB - * this process held while one session was open. + * A position read names an INDEX, and an index is a position inside ONE body. The body the page + * is showing and the body the service would serve can be two different bodies — a transcript + * rewritten under a running page is the ordinary way that happens — and then the same index + * names two different records. The page says which one it means by sending the record's + * version; the service compares it with the version of the record it would serve, so the two + * outcomes are told apart instead of the newer body being served under the older body's number. * - * The digest is what makes that safe to do. Between building the body and answering this request - * the file can be rewritten; the trajectory entry is keyed on the file's size and mtime, so a - * rewrite that preserved both would otherwise be served the wrong message under the right - * record's name. Answering `raw_stale` says the session moved instead. + * TWO REFUSALS, not one: a read that does not say is a read the service cannot judge, and a + * read that says the wrong thing is a read about a record that is not there. They are different + * answers, so they are asked separately — and the page's retry can tell them apart. * - * A missing entry means this process restarted between the two calls; saying so is more honest - * than silently showing the normalized record in a panel labelled 原文. + * @param {URL} url + * @returns {{ status: number, code: string, message: string } | null} the refusal, or null + */ + const missingRecordVersion = (url) => { + const asked = url.searchParams.get('v'); + if (typeof asked === 'string' && asked !== '') return null; + return { status: 400, code: 'record_version_required', message: '缺少记录版本，请重新加载会话' }; + }; + + /** + * The refusal a position read gets when it names a record the service would not serve. + * + * @param {URL} url + * @param {{ digest?: string } | undefined} locator the record's line locator + * @param {number} rowPos the record's position inside that line + * @returns {{ status: number, code: string, message: string } | null} the refusal, or null */ + const mismatchedRecordVersion = (url, locator, rowPos) => { + const asked = url.searchParams.get('v'); + if (asked === recordVersionToken(locator ? locator.digest : null, rowPos)) return null; + return { status: 409, code: 'record_version_mismatch', message: '该记录所在的版本已变化，请重新加载会话' }; + }; + /** * One record's four bulk fields, read on request. * @@ -3858,7 +5446,14 @@ export async function start(context) { sendError(response, 400, 'record_index_required', '缺少记录序号'); return; } - const cached = trajectoryCache.get(ctx.cacheKey); + // The key names the version and the family names the question. A read that names the + // session by its id and a poll that asked for `latest` are the same reader of the same + // body, so the second one falls back to the family's entry instead of reporting that + // nothing is cached: the two computed different keys because one of them carried the + // binding it resolved and the other did not, and that has nothing to do with the body. + // The session is part of the question: a directory can stop holding one session and start + // holding another, and the old session's body is not an answer about the new one. + const cached = trajectoryCache.get(ctx.cacheKey) ?? trajectoryCache.getByFamily(ctx.family, ctx.identityId); if (!cached) { sendError(response, 503, 'trajectory_not_cached', '会话数据已释放，请重新加载'); return; @@ -3869,6 +5464,14 @@ export async function start(context) { sendError(response, 404, 'record_not_found', '找不到该记录'); return; } + // Which record of which version the page is asking about. The entry in the cache may be a + // newer body than the one the page is showing, and then the index names another record. + const refused = missingRecordVersion(url) ?? mismatchedRecordVersion(url, locator, rowPos); + if (refused) { + context.logger.warn('miniapp.trajectory.read_error', { reason: refused.code }); + sendError(response, refused.status, refused.code, refused.message); + return; + } const found = await readRecordRow(locator); if ('error' in found) { context.logger.warn('miniapp.trajectory.read_error', { reason: `record_${found.reason}` }); @@ -3898,6 +5501,10 @@ export async function start(context) { } sendJson(response, 200, { index: wanted, + // No version here. The version's job was to make the REQUEST unambiguous, and the service + // above either served the record that was asked for or refused: an answer that repeated the + // version would be a field the page never reads, and a field nobody reads is a promise with + // no owner. The body carries the version of every record, and the page sends it back. // No `kind`. Measured over 6568 records of this machine's store: re-projecting one row // returns the same four values every time, but it disagrees about `kind` for an opener — // a `steer` row projected alone has no earlier turn to have been interrupted by, so it @@ -3947,6 +5554,22 @@ export async function start(context) { return { row, sizeBytes: bytes.length }; }; + /** + * One record's untouched message, fetched only when something actually reads it. + * + * The body no longer has the parsed rows behind it, so the line is read back out of the file it + * came from. That is a seek and a digest over one line, against the alternative of holding every + * session's object graph in memory for the life of the page — measured at 233 MB of the 378 MB + * this process held while one session was open. + * + * The digest is what makes that safe to do. Between building the body and answering this request + * the file can be rewritten; the trajectory entry is keyed on the file's size and mtime, so a + * rewrite that preserved both would otherwise be served the wrong message under the right + * record's name. Answering `raw_stale` says the session moved instead. + * + * A missing entry means this process restarted between the two calls; saying so is more honest + * than silently showing the normalized record in a panel labelled 原文. + */ const handleRaw = async (response, url) => { const ctx = await resolveTrajectoryRequest(url); if (!ctx.ok) { @@ -3958,11 +5581,28 @@ export async function start(context) { sendError(response, 400, 'record_index_required', '缺少记录序号'); return; } - const cached = trajectoryCache.get(ctx.cacheKey); + // The key names the version and the family names the question — see `handleRaw` for why the + // family is the fallback, and why the session is part of it. + const cached = trajectoryCache.get(ctx.cacheKey) ?? trajectoryCache.getByFamily(ctx.family, ctx.identityId); if (!cached) { sendError(response, 503, 'trajectory_not_cached', '会话数据已释放，请重新加载'); return; } + const locator = cached.raws.get(wanted); + const rowPos = cached.rowPos.get(wanted); + if (locator === undefined || typeof locator.filePath !== 'string' || typeof rowPos !== 'number') { + sendError(response, 404, 'record_not_found', '找不到该记录'); + return; + } + // Which record of which version the page is asking about. The entry in the cache may be a + // newer body than the one the page is showing, and then the index names another record — and + // the decoded message kept below would be that other record's message. + const refused = missingRecordVersion(url) ?? mismatchedRecordVersion(url, locator, rowPos); + if (refused) { + context.logger.warn('miniapp.trajectory.read_error', { reason: refused.code }); + sendError(response, refused.status, refused.code, refused.message); + return; + } // Re-reading a record the reader has already opened is the common case, and a decoded message // is far smaller than a session, so the entry keeps the ones it has already paid for. const alreadyDecoded = cached.decoded.get(wanted); @@ -3970,11 +5610,6 @@ export async function start(context) { sendJson(response, 200, { index: wanted, raw: alreadyDecoded.value, omitted: alreadyDecoded.omitted }); return; } - const locator = cached.raws.get(wanted); - if (locator === undefined || typeof locator.filePath !== 'string') { - sendError(response, 404, 'record_not_found', '找不到该记录'); - return; - } const found = await readRecordRow(locator); if ('error' in found) { context.logger.warn('miniapp.trajectory.read_error', { reason: `raw_${found.reason}` }); @@ -4107,7 +5742,9 @@ export async function start(context) { trajectoryCache.clear(); // The parsed-row cache only ever holds sealed generations now, but it is still a cache that // has to be released here: a disposed runtime that kept it would still hold a rotated - // generation's rows. The line is here so the next cache added cannot repeat the omission. + // generation's rows. Clearing it also stops the sweeper that releases the rows nobody wants + // any more, so a disposed runtime leaves no timer behind either. The line is here so the next + // cache added cannot repeat the omission. parsedRows.clear(); clientEntry = null; disposal = close(server).then(() => undefined); @@ -4137,6 +5774,9 @@ function stripInternalFields(value) { // decides by whether this field is set, so a missing key and a present key both reading // "not set" is what keeps one row from being a branch and the next from merely lacking it. sessionParentId: value.sessionParentId ?? null, + // The same floor, for the same reason: this is the token the page turns into 「轻量」/「定时」/ + // 「任务」, and a row with no kind of its own has to be able to say so. + sessionTag: value.sessionTag ?? null, }; } diff --git a/plugins/avatasia/mmc-trajectory/tests/client-check.mjs b/plugins/avatasia/mmc-trajectory/tests/client-check.mjs index ec4c3ccf..82ce653d 100644 --- a/plugins/avatasia/mmc-trajectory/tests/client-check.mjs +++ b/plugins/avatasia/mmc-trajectory/tests/client-check.mjs @@ -7,6 +7,10 @@ import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { execFileSync } from 'node:child_process'; import vm from 'node:vm'; +// The service's own fold over a set of records. The page has a copy of it — two copies of one +// question is two answers, and the difference between them is nobody's to explain — so the two are +// compared here rather than trusted. +import { recordSetFingerprint } from '../miniapp/node/server.mjs'; const HTML = fileURLToPath(new URL('../miniapp/client/index.html', import.meta.url)); const SERVER = fileURLToPath(new URL('../miniapp/node/server.mjs', import.meta.url)); @@ -227,7 +231,11 @@ check('no kinds match', sandbox.hasKind({ kind: 'system' }, ['user']), false); const netCtx = { console, AbortController, setTimeout, clearTimeout, encodeURIComponent }; vm.createContext(netCtx); const needed = ['function isCount(value)', 'function formatInt(value)', 'function searchHitSet(hits)', - 'function sessionParam()', 'function searchStatusLabel(searchState)', 'function runSearch(needle)']; + 'function sessionParam()', 'function displayedSessionParam()', 'function displayedGeneration()', + 'function displayedRecordsByIndex()', 'function displayedRecordVersions()', + 'function displayedRecordSet(versions)', 'function recordSetFingerprint(versions)', + 'function searchStatusLabel(searchState)', + 'function searchAnswerDoesNotApply()', 'function runSearch(needle)']; const sources = needed.map((sig) => fnSource(sig)); check('the shipped search functions are all still findable by signature', needed.filter((sig, i) => !sources[i]), []); @@ -237,7 +245,18 @@ check('no kinds match', sandbox.hasKind({ kind: 'system' }, ['user']), false); // page's state is, because nothing else about the page's state is what is under test. 'var state = { query: "retry me", sessionId: null, hostSessionId: null, followLatest: true,', ' generation: null, searchHits: null, searchTotal: 0, searchTruncated: false,', - ' searchPending: false, searchError: null };', + ' searchPending: false, searchError: null, searchRefresh: 0, searchRefreshFor: "",', + ' searchSet: "",', + // The body on screen. `displayedSessionParam()` reads it and the answer is checked against + // it, so a harness with no body on screen cannot ask either question. Its `generation` is + // the slice the body IS: null is the whole lineage, a number is that one generation. + // + // `flat` is the same body's records, which is what a hit is checked against: a hit names a + // POSITION, and the position only names a record if the version on both sides agrees. + ' flat: [ { turn: {}, record: { index: 1, v: "version-one" } },', + ' { turn: {}, record: { index: 5, v: "version-five" } },', + ' { turn: {}, record: { index: 7, v: "version-seven" } } ],', + ' payload: { session: { id: "mvs_displayed", generation: null } } };', 'var searchSeq = 0;', 'var searchController = null;', // The two renderers paint the page; neither is what is being tested and neither has a DOM here. @@ -249,19 +268,40 @@ check('no kinds match', sandbox.hasKind({ kind: 'system' }, ['user']), false); + ' is not in index.html any more); }')), ].join('\n'), netCtx); - const settle = async () => { + const settle = async (turns = 3) => { // fetch -> response.json() -> then -> then -> then is all microtasks plus the json() promise. // Three macrotask turns drain all of them, so the count does not have to be right. - for (let i = 0; i < 3; i += 1) await new Promise((done) => setImmediate(done)); + for (let i = 0; i < turns; i += 1) await new Promise((done) => setImmediate(done)); }; - netCtx.fetch = () => Promise.reject(Object.assign(new Error('api'), { code: 'search_failed' })); + // The page re-asks against the body it fetches when an answer turns out to be about another + // version of the records. The fetch itself is not what is under test here, so it is a stub that + // counts the calls: how many times the page was willing to chase the answer IS the behaviour. + netCtx.refreshes = 0; + netCtx.fetchTrajectory = function () { + netCtx.refreshes += 1; + return Promise.resolve(); + }; + + // Every URL the search asked for, so the question it asked can be read off the requests + // rather than out of the source. + const asked = []; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.reject(Object.assign(new Error('api'), { code: 'search_failed' })); + }; netCtx.runSearch('retry me'); await settle(); const afterFailure = netCtx.searchStatusLabel(netCtx.state); - netCtx.fetch = () => Promise.resolve({ ok: true, status: 200, - json: () => Promise.resolve({ hits: [{ index: 7 }], total: 1, truncated: false }) }); + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 7, v: 'version-seven' }], total: 1, truncated: false, + // The server names the session it answered for. That name is what the page checks the + // answer against before it lets the hits filter the body on screen. + session: { id: 'mvs_displayed', generation: null } }) }); + }; netCtx.runSearch('retry me'); await settle(); @@ -271,6 +311,394 @@ check('no kinds match', sandbox.hasKind({ kind: 'system' }, ['user']), false); check('so the label reports the answer instead of the stale failure', netCtx.searchStatusLabel(netCtx.state), '命中 1 条'); check('and the hit set it describes is the new one', [...netCtx.state.searchHits], [7]); + + /* The search has to be about the body on screen, and the answer has to say so. + * + * The search used to ask for `latest`, which is a different question from 「what is on + * screen」: the runtime can make another conversation current while the page still shows + * this one, and the answer then describes another session's records. The hits are indexes, + * so they FILTER this body's rows against another body's matches — a row that matches + * reads as a miss, and a row that does not read as a hit. Both are wrong in opposite + * directions, which is why the number of hits alone never showed the defect. + * + * Two things have to hold and neither one is enough: + * + * - the request names the displayed session, and + * - the answer is checked against it. + * + * A request that names the displayed session can still be answered about another one if + * the runtime moved in between, and an answer that is checked can still be about another + * session if the wrong question was asked. Both halves are asserted below. + */ + check('the search asks about the session on screen', + /[?&]session=mvs_displayed(&|$)/.test(asked[1]), true); + check('and it does not ask about whatever is current now', + /[?&]session=latest(&|$)/.test(asked[1]), false); + + // The answer comes back about another session. The hits must not be installed: they are + // indexes into another body's records, and installing them would report this body's rows + // against another body's matches. + const installed = [...netCtx.state.searchHits]; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 99 }], total: 1, truncated: false, + session: { id: 'mvs_other', generation: null } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('an answer about another session does not install its hit set', + [...netCtx.state.searchHits], installed); + check('and it does not report the other session\'s count either', + netCtx.state.searchTotal, 1); + check('it says the search failed, so the reader is told there is no answer', + netCtx.state.searchError, 'session_mismatch'); + check('and the label shows the failure rather than the hit count', + netCtx.searchStatusLabel(netCtx.state), '搜索失败，以下为上次结果'); + check('and the request it made still named the session on screen', + /[?&]session=mvs_displayed(&|$)/.test(asked[0]), true); + + // The same session, but another slice of it. A slice is a different body, so an answer + // about one slice is not an answer about another. + // + // The slice on screen is the body the page is SHOWING, not the generation the reader picked. + // The two come apart when the load of a picked generation fails: the selection says one thing + // and the body on screen says another, and only the body's own slice is the question the + // search may ask and answer. + netCtx.state.payload = { session: { id: 'mvs_displayed', generation: 3 } }; + netCtx.state.generation = 3; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 98 }], total: 1, truncated: false, + session: { id: 'mvs_displayed', generation: 2 } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('an answer about another slice of the same session does not install its hit set', + [...netCtx.state.searchHits], installed); + check('and the request named the slice that is on screen', + /[?&]generation=3(&|$)/.test(asked[0]), true); + check('and it failed for the same reason', + netCtx.state.searchError, 'session_mismatch'); + + // The reader picks another generation and the load of it FAILS. The body on screen is still + // the old one; the selection is the new one. A search that asked about the selection would be + // asking about a body that is not there, and the answer it got back would carry indexes of + // THAT body — which then filtered the rows the reader is actually looking at. + netCtx.state.payload = { session: { id: 'mvs_displayed', generation: 0 } }; + netCtx.state.generation = 1; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 1, v: 'version-one' }], total: 1, truncated: false, + session: { id: 'mvs_displayed', generation: 0 } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('the search asks about the slice on screen, not the one the reader picked', + /[?&]generation=0(&|$)/.test(asked[0]), true); + check('and it does not ask about the selection', + /[?&]generation=1(&|$)/.test(asked[0]), false); + check('so the answer about the body on screen is installed', + [...netCtx.state.searchHits], [1]); + check('and there is no failure, because the answer and the body agree', + netCtx.state.searchError, null); + + // The other direction, because a check that accepts every answer is not a check either. An + // answer about the SELECTED slice is not an answer about the body on screen, and its indexes + // must not filter that body. + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 42 }], total: 1, truncated: false, + session: { id: 'mvs_displayed', generation: 1 } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('an answer about the selected slice does not install its hit set', + [...netCtx.state.searchHits], [1]); + check('and it says the answer was about another body', + netCtx.state.searchError, 'session_mismatch'); + + // The other direction, because a check that rejects every answer is not a check. The page + // follows the runtime to another conversation, and the answer about THAT conversation is + // the one that belongs to the body on screen now. + netCtx.state.payload = { session: { id: 'mvs_displayed_b', generation: null } }; + netCtx.state.generation = null; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 5, v: 'version-five' }], total: 1, truncated: false, + session: { id: 'mvs_displayed_b', generation: null } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('the answer about the conversation the page moved to is installed', + [...netCtx.state.searchHits], [5]); + check('and there is no failure, because this time there is an answer', + netCtx.state.searchError, null); + check('and the request asked about the conversation on screen at that moment', + /[?&]session=mvs_displayed_b(&|$)/.test(asked[0]), true); + + /* The version. A hit is a POSITION, and the same position in two bodies can be two different + * records: a transcript rewritten under the page leaves the indexes alone and changes the + * records behind them. The session and the slice say WHICH BODY the answer is about; they say + * nothing about whether the record at a given index is still the same record. + * + * So an answer about the rewritten body would light up a row whose text does not contain the + * needle at all — the page would be saying 「命中 1 条」 about a row that does not match. The + * version on the hit and the version on the body have to agree before a hit may filter + * anything. */ + const beforeVersion = [...netCtx.state.searchHits]; + netCtx.state.payload = { session: { id: 'mvs_displayed', generation: null } }; + netCtx.state.generation = null; + netCtx.state.searchRefresh = 0; + netCtx.state.searchRefreshFor = ''; + netCtx.refreshes = 0; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ hits: [{ index: 7, v: 'a-newer-version' }], total: 1, truncated: false, + session: { id: 'mvs_displayed', generation: null } }) }); + }; + netCtx.runSearch('retry me'); + await settle(24); + check('a search answer for a newer body cannot filter the displayed older body', + [...netCtx.state.searchHits], beforeVersion); + check('and the page asks again against the body it fetches, a bounded number of times', + netCtx.refreshes, 2); + check('and then it says the answer does not apply to what is on screen', + netCtx.state.searchError, 'search_version_mismatch'); + check('and the label shows the failure rather than a count it cannot stand behind', + netCtx.searchStatusLabel(netCtx.state), '搜索失败，以下为上次结果'); + + // A hit about a record this body does not have — the session grew after the body was built. + // It cannot be shown, so it is not counted either: the number the reader sees has to be the + // number of rows that are lit up. + netCtx.state.searchRefresh = 0; + netCtx.state.searchRefreshFor = ''; + netCtx.refreshes = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ + hits: [{ index: 7, v: 'version-seven' }, { index: 4242, v: 'a-record-not-on-screen' }], + total: 2, truncated: false, + session: { id: 'mvs_displayed', generation: null } }) }); + }; + netCtx.runSearch('retry me'); + await settle(); + check('a hit about a record that is not on screen is not installed', + [...netCtx.state.searchHits], [7]); + check('and the count is the number of rows that are lit up', + netCtx.state.searchTotal, 1); + check('and no failure is reported, because that answer did apply', + netCtx.state.searchError, null); + check('and the page did not go chasing a body it did not need to', + netCtx.refreshes, 0); + + /* WHICH SET the answer is about. + * + * A search answer is about a SET of records: it says 「these match and the rest do not」, and + * the part that matters to the reader is the part it does NOT name. So the request has to say + * which set — how many records the page has, and a fingerprint of their versions — and the + * service has to answer only if it searched THAT set. Without it, an answer about another body + * dims a record the reader can see, and says nothing about it. + */ + check('the search names how many records the body on screen has', + /[?&]n=3(&|$)/.test(asked[0]), true); + check('and a fingerprint of their versions', + /[?&]fp=[0-9a-f]{16}(&|$)/.test(asked[0]), true); + check('and the page folds a set of records into the same answer the service does', + netCtx.recordSetFingerprint(['version-one', 'version-five', 'version-seven']), + recordSetFingerprint(['version-one', 'version-five', 'version-seven'])); + check('and a set that is not the same is not the same answer', + netCtx.recordSetFingerprint(['version-one', 'version-five']) + === netCtx.recordSetFingerprint(['version-one', 'version-seven']), false); + check('and the count is appended, so the request still names the session first', + /[?&]session=mvs_displayed(&|$)/.test(asked[0]), true); + + // The service refuses: the body it searched is not the body on screen. That is the same thing a + // hit with the wrong version says, and it has to be answered the same way — the answer is not + // installed, the body is fetched once more, and the same question is put again against it. + const beforeRefusal = [...netCtx.state.searchHits]; + netCtx.state.searchRefresh = 0; + netCtx.state.searchRefreshFor = ''; + netCtx.refreshes = 0; + asked.length = 0; + netCtx.fetch = (url) => { + asked.push(String(url)); + return Promise.resolve({ ok: false, status: 409, + json: () => Promise.resolve({ error: 'search_version_mismatch', message: 'x' }) }); + }; + netCtx.runSearch('retry me'); + await settle(24); + check('an answer the service refuses is not installed', + [...netCtx.state.searchHits], beforeRefusal); + check('and the page asks again against a body it fetches, a bounded number of times', + netCtx.refreshes, 2); + check('and then it says the answer does not apply to what is on screen', + netCtx.state.searchError, 'search_version_mismatch'); + check('and the label shows the failure rather than a count it cannot stand behind', + netCtx.searchStatusLabel(netCtx.state), '搜索失败，以下为上次结果'); +} + +/* The answer has to be about the records that are ON SCREEN — and a body can change without + * changing its shape. + * + * The change detector is a cheap one, on purpose: it compares structural counters, because + * rebuilding 200+ rows every 2.5s is wasteful when nothing changed. But a record's TEXT can be + * rewritten under the page while every one of those counters stays the same — the same message + * count, the same turn count, the same timestamps, the same usage. The body on screen is then a + * different SET of records, and the answer standing on screen is about the old ones: the reader + * is told 「命中 1 条」 about a row that does not contain the needle, and nothing anywhere says + * so. + * + * The request side already knows this — it names the set it is asking about, and the service + * refuses a search about another set. But a refusal only happens if a REQUEST is made, and here + * none is: the answer is not re-asked, it is simply left standing. + * + * So this runs the real `applyPayload` with a body that was rewritten in place, and asks what + * the page does about it. */ +{ + const applyCtx = { console, AbortController, setTimeout, clearTimeout, encodeURIComponent }; + vm.createContext(applyCtx); + const needed = ['function isCount(value)', 'function signatureOf(payload)', 'function flattenTurns(turns)', + 'function payloadCursor(payload)', 'function deltaMatchesNumbering(previous, delta)', + 'function recordSetFingerprint(versions)', 'function displayedRecordVersions()', + 'function displayedRecordSet(versions)', 'function displayedSessionParam()', + 'function displayedGeneration()', 'function applyPayload(body, manual)']; + const sources = needed.map((sig) => fnSource(sig)); + check('the shipped functions the answer-set rule needs are all still findable by signature', + needed.filter((sig, i) => !sources[i]), []); + vm.runInContext([ + 'var turnById = new Map();', + 'var state = { payload: null, flat: [], signature: "", loading: false, hardError: null,', + ' since: null, sinceFor: null, etag: "", etagFor: "", query: "", searchSet: "" };', + // The re-ask is what is under test, so it is recorded rather than performed — and the set is + // recorded the way the shipped `runSearch` records it, because the page keeps answering + // about the set it last asked about. + 'var asked = [];', + 'function runSearch(needle) { asked.push(String(needle)); state.searchSet = displayedRecordSet(); }', + // The page paints itself; none of that is what is under test and none of it has a DOM here. + 'function render() {}', + 'function announce() {}', + 'function noteFresh() {}', + 'function rePinWindow() {}', + ...sources.map((src, i) => src || (needed[i] + ' { throw new Error(' + JSON.stringify(needed[i]) + + ' is not in index.html any more); }')), + ].join('\n'), applyCtx); + + // One record, and the body that holds it. The SHAPE is the same in every one of these bodies: + // the same session, the same counters, the same turn, the same one record. Only the record's own + // version token moves — and that is exactly what a rewrite in place looks like from here. + const bodyWith = (version, stats) => ({ + session: { id: 'mvs_displayed', generation: null }, + stats: Object.assign({ messages: 1, toolCalls: 0, errors: 0, durationMs: 1000, + tokens: { totalTokens: 10 } }, stats || {}), + turns: [{ id: 'turn-1', records: [{ index: 1, v: version, summary: 'ALPHA BODY' }] }] + }); + + // The body on screen, and a question that was answered for it. + const alpha = bodyWith('the-version-of-alpha'); + applyCtx.state.payload = alpha; + applyCtx.state.flat = applyCtx.flattenTurns(alpha.turns); + applyCtx.state.signature = applyCtx.signatureOf(alpha); + applyCtx.state.query = 'alpha'; + applyCtx.state.searchSet = applyCtx.displayedRecordSet(); + applyCtx.asked.length = 0; + + // The same body again: nothing moved, so the question is not put again. + applyCtx.applyPayload(bodyWith('the-version-of-alpha'), false); + check('a body that did not change does not put the question again', applyCtx.asked, []); + + // The record's text is rewritten in place. Every counter is the same; only the record's own + // version token is different, so the body on screen is a different SET of records. + applyCtx.applyPayload(bodyWith('the-version-of-charlie'), false); + check('a content-only payload change retires the previous search answer', + applyCtx.asked, ['alpha']); + + // And it is retired once, not on every poll afterwards. + applyCtx.asked.length = 0; + applyCtx.applyPayload(bodyWith('the-version-of-charlie'), false); + check('and the question is not put again on every poll after that', + applyCtx.asked, []); + + // The rule that was already there: a body whose SHAPE changed puts the question again, even + // when its records are the same ones. A change of counters is a change of body, and the answer + // on screen was computed against the old one. + applyCtx.asked.length = 0; + applyCtx.applyPayload(bodyWith('the-version-of-charlie', { durationMs: 2000 }), false); + check('a payload that changed re-runs the search rather than filtering by the old answer', + applyCtx.asked, ['alpha']); +} + +/* handoff: the session the record came from, across a navigation. + * + * The handoff reads two things asynchronously — the record's raw message and its four bulk + * fields — and the page can be showing another conversation by the time they answer. The text + * that goes to the Host NAMES the session, so a session read at the END of the reads would put + * one conversation's name on another conversation's message: the reader would be told which + * conversation they are looking at, and told wrong. + * + * This runs the shipped `handoffSelected` over a deferred raw read and moves the page while + * that read is in flight, which is the only moment the two can differ. */ +{ + const handoffSources = { + handoff: fnSource('function handoffSelected()'), + selected: fnSource('function selectedRecord()'), + bridge: fnSource('function hostBridge()'), + }; + check('the shipped handoff functions are all still findable by signature', + Object.keys(handoffSources).filter((key) => !handoffSources[key]), []); + + if (handoffSources.handoff && handoffSources.selected && handoffSources.bridge) { + const handoffCtx = { console, setTimeout, clearTimeout }; + vm.createContext(handoffCtx); + // The page's own pure helpers, so the markdown is the shipped one and not a copy of it. + vm.runInContext(block, handoffCtx); + const appended = []; + // The Host bridge, so `hostBridge()` finds a message sink without being stubbed away. + handoffCtx.window = { mcode: { miniapp: { message: { append: (text) => { appended.push(text); } } } } }; + handoffCtx.dom = { handoffBtn: { disabled: false }, inspectorStatus: { textContent: '' } }; + handoffCtx.state = { + selectedId: 'rec-1', + flat: [{ + id: 'rec-1', + record: { id: 'rec-1', index: 1, kind: 'user', role: 'user', timestamp: 1791600000000 }, + turn: { id: 't1', index: 3 }, + }], + // The conversation the selected record belongs to. + payload: { session: { id: 'mvs_a', label: 'A 会话' } }, + }; + let releaseRaw = null; + handoffCtx.rawSourceFor = () => new Promise((resolve) => { releaseRaw = resolve; }); + handoffCtx.fetchDetail = () => Promise.resolve(null); + handoffCtx.inspectorSay = () => {}; + handoffCtx.stripVolatile = (record) => record; + vm.runInContext( + handoffSources.handoff + '\n' + handoffSources.selected + '\n' + handoffSources.bridge, + handoffCtx, + ); + + handoffCtx.handoffSelected(); + // The page moves to another conversation while the record's raw read is still in flight. + handoffCtx.state.payload = { session: { id: 'mvs_b', label: 'B 会话' } }; + if (releaseRaw) releaseRaw({ raw: '{"kind":"user"}', omitted: false }); + for (let i = 0; i < 5; i += 1) await new Promise((done) => setImmediate(done)); + + check('the handoff names the session the selected record came from', + appended.length === 1 && appended[0].includes('mvs_a'), true); + check('and not the conversation the page moved to while the read was pending', + appended.length === 1 && appended[0].includes('mvs_b'), false); + } } /* flatten + signature */ @@ -527,47 +955,105 @@ const codeOnly = html .replace(/\/\*[\s\S]*?\*\//g, '') .replace(/^[ \t]*\/\/.*$/gm, ''); -/* ------------------------------------------------------------------ session picker: branch marker */ +/* ------------------------------------------------------------------ session picker: the two picks */ // A session that the runtime spawned from another one used to be listed exactly like a top-level // conversation, because nothing in the list said otherwise: the parent column existed in the store // and was never read, and the one field that did reach the page (`sessionKind`) had no reader in // it at all. A reader comparing two entries could not tell a sub-agent run from a conversation. // -// The marker is checked here by RUNNING it rather than by matching its text, because the failure -// worth catching is not a typo in the string — it is the marker being applied to everything, or to -// nothing, and both of those still contain the right characters. The decisive case is the third -// one: two sessions identical except for `sessionParentId`, whose labels must differ by exactly -// the marker and nothing else. A version that appended the marker unconditionally passes every -// "does it contain 子会话" check and fails this one. -const branchMarkSrc = /var BRANCH_MARK = '[^']*';/.exec(html); -const branchLabelSrc = fnSource('function sessionOptionLabel(session, now)'); -if (!branchMarkSrc || !branchLabelSrc) { - failures.push('session picker: could not lift sessionOptionLabel / BRANCH_MARK'); +// The observation still stands: `sessionKind` is sent and still has no reader. What the reader now +// gets instead is `sessionTag` — the token the service derives from the row's purpose and kind — +// and it is the one the page draws. The raw kind column is still on the wire and still unread. +// +// How a run is told apart is now WHICH CONTROL it is in, not what its label says: the conversation +// pick holds conversations, and the run pick holds the runs of the conversation that was picked. +// The label says nothing about it — one fact, one signal — and the signal is the control itself. +const labelSrc = fnSource('function sessionOptionLabel(session, now)'); +if (!labelSrc) { + failures.push('session picker: could not lift sessionOptionLabel'); } else { - vm.runInContext(branchMarkSrc[0] + '\n' + branchLabelSrc, sandbox); + vm.runInContext(labelSrc, sandbox); const NOW = 1_800_000_000_000; const base = { label: '一个会话', lastActiveAt: NOW - 60_000, messageCount: 42 }; const plain = sandbox.sessionOptionLabel({ ...base }, NOW); - const branch = sandbox.sessionOptionLabel({ ...base, sessionParentId: 'mvs_parent' }, NOW); + const run = sandbox.sessionOptionLabel({ ...base, sessionParentId: 'mvs_parent' }, NOW); - assert('a top-level session is not marked as a branch', + // The label does not say it — the control does. A version that put the mark back into the + // label would say it twice, and a version that dropped it without moving the row would say it + // not at all. + assert('a run is not marked in its label', + run === plain); + assert('and a conversation is not marked either', plain.indexOf('子会话') < 0); - assert('a session the runtime spawned is marked as one', - branch.indexOf('子会话') >= 0); - // Identity, not shape: the only thing that differs between these two inputs is the field, so - // the only thing allowed to differ between the two labels is the marker. - assert('the marker is driven by the field, not by where the row sits in the list', - branch.replace(' · 子会话', '') === plain); - assert('and nothing else about the label changes with it', - branch.length === plain.length + ' · 子会话'.length); - // An empty parent string is "no parent", not "a parent named the empty string". + // An empty parent id is "no parent", not "a parent named the empty string". assert('an empty parent id is not a parent', sandbox.sessionOptionLabel({ ...base, sessionParentId: '' }, NOW) === plain); - // The unnamed branch takes the marker too — otherwise the one case with no title to read is - // the one case a reader cannot place. - assert('an unnamed branch is still marked', - sandbox.sessionOptionLabel({ lastActiveAt: NOW - 60_000, messageCount: 42, - sessionParentId: 'mvs_parent' }, NOW).indexOf('子会话') >= 0); +} + +/* ------------------------------------------------------------------ session picker: kind badge */ +// The other half of the same defect: a cron run, a background task and a lightweight exchange were +// listed exactly like the reader's own conversations, because the service read the runtime +// database's `purpose` column not at all — so it had no way to tell them apart, and a row that +// looked like a conversation was presented as one. +// +// The badge is driven by a token the service derived, never by the raw purpose: that value is free +// text and can carry anything a person typed into it, so publishing it would hand the reader's +// browser somebody else's words. The token set is closed, and a token this page does not know +// about produces no badge rather than one this page invented. +// +// Same discipline as the layer check above, and for the same reason: the failure worth catching +// is not a typo in the string, it is the badge being applied to everything or to nothing, and both +// of those still contain the right characters. +const tagBadge = sandbox.sessionTagBadge; +assert('the badge helper is in the pure block, so a label can be built without a DOM', + typeof tagBadge === 'function'); +if (typeof tagBadge === 'function') { + check('a lightweight session is badged 轻量', tagBadge('lightweight'), ' · 轻量'); + check('a scheduled session is badged 定时', tagBadge('scheduled'), ' · 定时'); + check('a background task is badged 任务', tagBadge('task'), ' · 任务'); + check('no kind at all is no badge', tagBadge(null), ''); + check('an empty token is no badge', tagBadge(''), ''); + check('a token this page does not know is no badge', tagBadge('something-else'), ''); + + const NOW = 1_800_000_000_000; + const base = { label: '一个会话', lastActiveAt: NOW - 60_000, messageCount: 42 }; + const plain = sandbox.sessionOptionLabel({ ...base }, NOW); + const light = sandbox.sessionOptionLabel({ ...base, sessionTag: 'lightweight' }, NOW); + + assert('a lightweight session is badged in the picker', light.indexOf('轻量') >= 0); + assert('a scheduled session is badged in the picker', + sandbox.sessionOptionLabel({ ...base, sessionTag: 'scheduled' }, NOW).indexOf('定时') >= 0); + assert('a background task is badged in the picker', + sandbox.sessionOptionLabel({ ...base, sessionTag: 'task' }, NOW).indexOf('任务') >= 0); + // The three "no badge" cases, which are three different ways of saying the same thing. + assert('a session with no kind of its own carries no badge', + sandbox.sessionOptionLabel({ ...base, sessionTag: null }, NOW) === plain); + assert('an empty token carries no badge', + sandbox.sessionOptionLabel({ ...base, sessionTag: '' }, NOW) === plain); + assert('a row with no kind field at all carries no badge', + sandbox.sessionOptionLabel({ ...base }, NOW) === plain); + // Identity, not shape: the only thing that differs between these two inputs is the field, so + // the only thing allowed to differ between the two labels is the badge. + assert('the badge is driven by the field, not by where the row sits in the list', + light.replace(' · 轻量', '') === plain); + assert('and nothing else about the label changes with it', + light.length === plain.length + ' · 轻量'.length); + // Where the badge sits inside the label: the name first, then the times and the count, then + // what the session IS. Moved to the front, the row still carries the right characters in the + // right places — every check that asks 「does the row say 轻量」 passes — and the row now reads + // as a category with a name attached to it. + assert('the badge comes after the name, not in front of it', + light.indexOf(' · 轻量') > light.indexOf(base.label), + JSON.stringify(light)); + // Both facts at once: the kind says what the session IS, and the parent says where it came + // from. The kind is in the label; the parent is in the position, so the label carries the kind + // and no longer says 「子会话」 in words — one fact, one signal. + const both = sandbox.sessionOptionLabel({ ...base, sessionTag: 'lightweight', sessionParentId: 'mvs_parent' }, NOW); + assert('a row with both a kind and a parent still shows the kind', + both.indexOf(' · 轻量') >= 0); + assert('and no longer claims to be a branch in words — where it sits says that', + both.indexOf('子会话') < 0, + both); } const checks = [ @@ -969,21 +1455,19 @@ const checks = [ const gates = src.match(/if \(seq !== searchSeq\) return;/g) || []; return gates.length === 2 && /var seq = \(searchSeq \+= 1\);/.test(src); })()], - ['a payload that changed re-runs the search rather than filtering by the old answer', (() => { - const src = fnSource('function applyPayload(body, manual)'); - if (!src) return false; - return /if \(changed && state\.query\) runSearch\(state\.query\);/.test(src); - })()], ['clearing the filters drops the answer with them', (() => { const src = fnSource('function resetBaseFilters()'); if (!src) return false; return /state\.searchHits = null;/.test(src) && /state\.searchError = null;/.test(src); })()], - ['a search is asked about the slice the reader is looking at', (() => { + ['a search is asked about the slice the body on screen is', (() => { const src = fnSource('function runSearch(needle)'); if (!src) return false; return /'\/api\/trajectory\/search\?session='/.test(src) - && /&generation=' \+ encodeURIComponent\(String\(state\.generation\)\)/.test(src); + && /&generation=' \+ encodeURIComponent\(String\(generation\)\)/.test(src) + // The slice the reader PICKED is a different question, and the page has to ask the one + // that belongs to the body it is showing. + && !/String\(state\.generation\)/.test(src); })()], // Clicking the window button must not move the page. It scrolled to the document tail // whenever it narrowed the window, on the theory that "back to the tail" should land on the @@ -1039,15 +1523,35 @@ const checks = [ // A checkpoint is drawn as a band, not as a row. It has to be routed there in the one place // the ledger builds its list — routing it anywhere else would leave the row grid laying it out // like ordinary work while the CSS sits unused. + // + // That place is fillGroupRows(), which the build calls: the rows of a group are built by the + // build and by the observer that fills a group the reader scrolls to, and a second copy of the + // loop would be two definitions of what a row is. Both halves are checked — a routing rule in a + // function the ledger does not call is a rule about nothing. ['a compaction record renders as a band, not as a row', (() => { if (!/function buildCompactionBand\(entry\)/.test(html)) return false; + const fill = fnSource('function fillGroupRows(list, group)'); + if (!fill) return false; const render = fnSource('function renderLedger()'); if (!render) return false; - const loop = render.slice(render.indexOf('var rails = railStates(')); - if (!loop) return false; - return /group\.entries\[j\]\.record\.kind === 'compaction'/.test(loop) - && /item\.className = 'compaction-cell'/.test(loop) - && /item\.appendChild\(buildCompactionBand\(group\.entries\[j\]\)\)/.test(loop); + // The test and the branch it decides have to be one thing. + // + // Checking that the word 'compaction' appears in the function is not the same as checking + // that a compaction record is rendered as a band: a version that computes the test and then + // ignores it says the same word and renders the record as a row. That is not a hypothetical + // — the two were once the same line, and splitting them (the test into a variable, the + // branch into an `if`) left the old assertion checking a word rather than a decision. + // + // So the band branch is taken out of the function and read on its own, and the branch has to + // be the one the test opens. The row branch is checked too: a version that renders EVERY + // record as a band has the band branch right and is wrong in the other direction. + const band = fill.match(/if \(isBand\) \{([\s\S]*?)\n {10}\} else \{/); + return /var isBand = group\.entries\[j\]\.record\.kind === 'compaction';/.test(fill) + && !!band + && /item\.className = 'compaction-cell'/.test(band[1]) + && /item\.appendChild\(buildCompactionBand\(group\.entries\[j\]\)\)/.test(band[1]) + && /item\.appendChild\(buildRow\(group\.entries\[j\], rails\[j\]\)\)/.test(fill) + && /fillGroupRows\(holds \? list : null, group\)/.test(render); })()], // The band is a real control: it carries the same data-row-id the inspector looks up, so // clicking it opens the checkpoint's details rather than doing nothing. The two halves are @@ -1150,6 +1654,756 @@ for (const [name, ok] of checks) { else failures.push('policy: ' + name); } +/* ------------------------------------------------------------------ session picks: the cascade */ +// Two controls now. The conversation pick names the conversation; the run pick names the run +// inside it, and holds the runs of the conversation the first pick named. A run used to sit in +// the conversation pick as a peer of the conversations — thirteen of twenty rows on this machine +// were runs of the one conversation being read — and it used to sit there indented under its +// conversation, which made the dropdown forty rows long. +// +// The failures worth catching are not strings that survived an edit: they are a run sitting in +// the conversation pick, a run pick that is empty when it should not be, a run pick that lists +// the runs of the wrong conversation, and a pick that does not switch. So everything below is +// read back out of a rendered picker, not out of its source. +// +// The functions are lifted HERE, before the picker section below, because that section runs +// `renderSessionSelect` too and the picker now asks the same question this section asks: which +// rows are runs, and which control holds them. +const layerFns = [ + 'function sessionOptionLabel(session, now)', + 'function childrenByParent()', + 'function ownerSessionId()', + 'function renderSessionSelect()', + 'function renderChildSelect()', + 'function selectSession(id)', + 'function setFollow(enabled)', + 'function onSessionSelectChange()', + 'function onChildSelectChange()' +]; +const layerSrc = layerFns.map((name) => fnSource(name)); +if (layerSrc.some((src) => !src)) { + failures.push('session layers: could not lift ' + layerFns.filter((_, i) => !layerSrc[i]).join(', ')); +} else { + // selectSession() drops what the ledger knows about the groups of the session it is leaving, so + // the two maps it uses have to exist here. They are the page's own, lifted as empty ones: what + // is under test is which session is chosen, not what the ledger remembers about it. + sandbox.ledgerState = new Map(); + sandbox.ledgerContent = new Map(); + vm.runInContext(layerSrc.join('\n'), sandbox); +} + +// A declaration rather than a const, because the picker section below uses it and this section +// is written after it: a `const` would be in its temporal dead zone there, and the failure would +// read as a broken stub rather than as an ordering mistake. +function stubSelect() { + const options = []; + return { + options, + selectedIndex: -1, + _value: '', + get value() { return this._value; }, + set value(next) { + this._value = next; + this.selectedIndex = options.findIndex((option) => option.value === next); + }, + get firstChild() { return options[0] || null; }, + removeChild(child) { + const at = options.indexOf(child); + if (at >= 0) options.splice(at, 1); + return child; + }, + appendChild(node) { + options.push(node); + if (this.selectedIndex < 0) this.selectedIndex = options.length - 1; + return node; + }, + insertBefore(node, reference) { + const at = reference ? options.indexOf(reference) : -1; + options.splice(at < 0 ? options.length : at, 0, node); + return node; + }, + }; +} + +/* ------------------------------------------------------------------ session picker: where the pinned row goes */ +// The picker can hold a row the server did not send: the session the reader pinned by hand, which +// aged out of the twenty the service returns. The page adds that row itself, and where it puts it +// is a promise about what the reader sees — the service's first row is the conversation the +// runtime database reports as current, and the pinned row is a different thing, so "the current +// conversation is first" does not by itself say where the pinned row lands. +// +// It is checked by RUNNING `renderSessionSelect` against a stub select, not by reading its +// source. A regex over the source would pass for a version that inserted the row in the right +// place for the wrong reason, and would not notice a version that inserted it in the right place +// only when the list happened to be empty. What the reader gets is the order of the options, so +// that is what is read back. +const renderSelectSrc = fnSource('function renderSessionSelect()'); +if (!renderSelectSrc) { + failures.push('session picker: could not lift renderSessionSelect'); +} else { + // A select, as far as this function touches it: the options it holds, the one that is + // selected, and the two ways a row can be added to it. The browser picks the first option + // when the list is built and picks by value afterwards, and `insertBefore` puts a row at a + // position rather than at the end — which is the whole point of the row in question. + sandbox.el = (tag, className, text) => { + const node = stubNode(tag, text); + if (className) node.className = className; + node.value = ''; + return node; + }; + sandbox.clear = (node) => { while (node && node.firstChild) node.removeChild(node.firstChild); }; + if (!sandbox.state) sandbox.state = {}; + sandbox.state.sessions = [ + { id: 'mvs_server_first', label: '服务端第一行', lastActiveAt: 1_800_000_000_000, messageCount: 3 }, + { id: 'mvs_server_second', label: '服务端第二行', lastActiveAt: 1_799_999_000_000, messageCount: 4 }, + ]; + sandbox.state.followLatest = false; + sandbox.state.sessionId = 'mvs_the_pinned_one'; + vm.runInContext(renderSelectSrc, sandbox); + + const select = stubSelect(); + sandbox.dom = { sessionSelect: select }; + sandbox.renderSessionSelect(); + + const texts = select.options.map((option) => option.textContent); + // The row the reader pinned is the one the service did not send, so it is on its own: the + // first row is 「跟随最新」 and the pinned row is the first of the session rows. + assert('the row the reader pinned is the first session row, above the rows the service sent', + texts.length === 4 + && texts[0].indexOf('跟随最新') >= 0 + && texts[1].indexOf('当前会话（不在最近列表）') >= 0 + && select.options[1].value === 'mvs_the_pinned_one', + texts.join(' | ')); + assert('and the service\'s own first row is still the one right after it', + select.options[2] && select.options[2].value === 'mvs_server_first', + texts.join(' | ')); + assert('and the pinned row is the one that ends up selected', + select.value === 'mvs_the_pinned_one' && select.selectedIndex === 1, + String(select.value) + ' @ ' + String(select.selectedIndex)); + + // And the other direction: a pinned session the service DID send needs no row of its own, or + // the reader would see the same conversation twice. + sandbox.state.sessionId = 'mvs_server_second'; + const second = stubSelect(); + sandbox.dom = { sessionSelect: second }; + sandbox.renderSessionSelect(); + assert('a pinned session the service sent is not added a second time', + second.options.length === 3 + && second.options.every((option, index) => option.value !== 'mvs_the_pinned_one') + && second.value === 'mvs_server_second', + second.options.map((option) => option.value).join(' | ')); +} + +/* ------------------------------------------------------------------ session picks: the cascade */ +// The functions were lifted above, before the picker section: the picker asks the same question +// this section asks, so it needs them too. +if (layerSrc.every((src) => src)) { + const NOW = 1_800_000_000_000; + const MAIN_A = 'mvs_main_a'; + const MAIN_B = 'mvs_main_b'; + // Thirteen of them, because that is the number this machine actually has: the list is not a + // window, and a fixture with three rows would not notice one that quietly dropped the tenth. + const childrenOfA = Array.from({ length: 13 }, (_, i) => ({ + id: 'mvs_child_a' + i, + sessionParentId: MAIN_A, + label: '子代理甲 ' + i, + lastActiveAt: NOW - (i + 1) * 1000, + messageCount: i + 1 + })); + const childrenOfB = [{ + id: 'mvs_child_b0', sessionParentId: MAIN_B, label: '子代理乙', + lastActiveAt: NOW - 3000, messageCount: 5 + }]; + const layerState = { + sessions: [ + { id: MAIN_A, label: '主会话甲', lastActiveAt: NOW, messageCount: 10 }, + { id: MAIN_B, label: '主会话乙', lastActiveAt: NOW - 60_000, messageCount: 20 } + ], + children: childrenOfA.concat(childrenOfB), + followLatest: true, + sessionId: MAIN_A, + hostSessionId: null + }; + sandbox.state = layerState; + // The two picks, as two stub selects: the conversation one and the run one. + const convo = stubSelect(); + const runs = stubSelect(); + sandbox.el = (tag, className, text) => { + const node = stubNode(tag, text); + if (className) node.className = className; + node.value = ''; + return node; + }; + sandbox.clear = (node) => { while (node && node.firstChild) node.removeChild(node.firstChild); }; + sandbox.dom = { + sessionSelect: convo, + childSelect: runs, + followBtn: { setAttribute: () => {} }, + followState: { textContent: '' } + }; + sandbox.fetchSessions = () => {}; + sandbox.fetchTrajectory = () => {}; + sandbox.announce = () => {}; + + sandbox.renderSessionSelect(); + sandbox.renderChildSelect(); + + // 1. While the page follows the newest conversation there is nothing to offer: that list + // changes under the reader, and a run picked out of it belongs to a conversation that is no + // longer on screen. + assert('the run pick is empty while the page follows the newest conversation', + runs.options.length === 1 && runs.options[0].value === '' && runs.value === '', + runs.options.map((option) => option.value).join(' | ')); + assert('and it is disabled, so there is nothing to pick', + runs.disabled === true, String(runs.disabled)); + + // 2. The conversation pick holds conversations and nothing else. This is the rule the whole + // feature exists for: a run must never sit where a conversation sits. + const convoValues = convo.options.map((option) => option.value); + assert('the conversation pick holds no run at all', + !convoValues.some((value) => /^mvs_child_/.test(value)), + convoValues.join(' | ')); + assert('and it holds every conversation', + convoValues.length === 3 + && convoValues.indexOf(MAIN_A) === 1 + && convoValues.indexOf(MAIN_B) === 2, + convoValues.join(' | ')); + + // 3. With a conversation picked, the run pick holds its runs — all of them, in the order the + // service sent them. + layerState.followLatest = false; + layerState.sessionId = MAIN_A; + sandbox.renderSessionSelect(); + sandbox.renderChildSelect(); + const aIds = childrenOfA.map((child) => child.id); + const runValues = runs.options.map((option) => option.value); + assert('the run pick holds every run of the conversation on screen', + runValues.length === 14, runValues.join(' | ')); + assert('and they keep the order the service sent', + runValues.slice(1).join(',') === aIds.join(','), runValues.join(' | ')); + assert('and the first option is the conversation itself', + runValues[0] === '' && (runs.options[0].textContent || '').indexOf('主会话本身') >= 0, + JSON.stringify(runs.options[0] && runs.options[0].textContent)); + assert('and nothing is picked while no run is picked', + runs.value === '', String(runs.value)); + + // 4. The two picks agree about what is on screen: a run belongs to a conversation, and the + // conversation pick says which one. Asked while a RUN is on screen -- asked while a + // conversation is on screen it would have the same answer either way, and a guard that + // cannot tell the two apart is not a guard. + layerState.sessionId = 'mvs_child_a5'; + assert('the owner of a run is the conversation it hangs off', + sandbox.ownerSessionId() === MAIN_A, sandbox.ownerSessionId()); + layerState.sessionId = MAIN_A; + + // 5. Picking a run switches the page to it — that run's own trajectory, since a run is a + // session with its own transcript — and leaves the conversation pick on its parent. + runs.value = 'mvs_child_a5'; + sandbox.onChildSelectChange(); + assert('picking a run switches the page to that run', + layerState.sessionId === 'mvs_child_a5' && layerState.followLatest === false, + layerState.sessionId + ' / follow=' + layerState.followLatest); + sandbox.renderSessionSelect(); + assert('and the conversation pick stays on its parent', + convo.value === MAIN_A, String(convo.value)); + + // 6. And picking the conversation itself returns to the conversation. + sandbox.renderChildSelect(); + runs.value = ''; + sandbox.onChildSelectChange(); + assert('picking the conversation itself returns to the conversation', + layerState.sessionId === MAIN_A, String(layerState.sessionId)); + + // 7. And picking another conversation moves the run pick with it, and clears the run. + convo.value = MAIN_B; + sandbox.onSessionSelectChange(); + sandbox.renderChildSelect(); + assert('picking another conversation moves the run pick with it', + runs.options.length === 2 && runs.options[1].value === 'mvs_child_b0', + runs.options.map((option) => option.value).join(' | ')); + assert('and clears the run that was picked', + runs.value === '', String(runs.value)); + + // 8. A parent id is data, not a key the page may assume. The runtime database's parent + // column crosses the wire unchanged, and the page groups its runs by that value. The + // names a plain object inherits -- '__proto__', 'constructor' -- are truthy and are not + // arrays, so a version that groups into a plain object reads one of them as a group and + // throws on it, and the picker stops updating after an otherwise successful list. + const hostile = ['__proto__', 'constructor', 'toString', 'valueOf']; + const realState = sandbox.state; + sandbox.state = { + sessions: [{ id: MAIN_A, label: '主会话甲', lastActiveAt: NOW, messageCount: 10 }], + children: hostile.map((parentId, index) => ({ + id: 'mvs_hostile_' + index, + sessionParentId: parentId, + label: '异常父 ' + index, + lastActiveAt: NOW - index, + messageCount: 1 + })), + followLatest: false, + sessionId: MAIN_A, + hostSessionId: null + }; + let grouped = null; + let groupingError = null; + try { + grouped = sandbox.childrenByParent(); + } catch (error) { + groupingError = error; + } + check('a parent id that names an inherited object property does not break the grouping', + groupingError === null && grouped !== null, true); + check('and those runs are not folded into a conversation that owns them', + grouped !== null + && Object.keys(grouped).every((key) => hostile.indexOf(key) >= 0) + && hostile.every((key) => !grouped[key] || grouped[key].length === 1), true); + let renderError = null; + try { + sandbox.renderChildSelect(); + } catch (error) { + renderError = error; + } + check('and the run pick still draws with such a parent on the page', renderError === null, true); + sandbox.state = realState; +} + +/* ------------------------------------------------------------------ session list: overlapping polls */ +// The list is polled, and every poll replaces the one before it. Two things have to hold +// while that happens, and both used to be false: +// +// * a request that has been replaced must not take the next request's controller with it; +// * an answer that arrives late must not replace a newer list. +// +// The first is what made the second possible: the replaced request cleared the shared +// controller on its way out, so the request that replaced it had nothing to abort, and the +// request before that answered last and won. +// +// So this section drives three overlapping requests against a controllable fetch, and +// reads back what the page ends up showing. It is the transition, not the shape of the +// source: a regex would pass for a version that carries the sequence number and ignores it. + +const fetchSessionsSrc = fnSource('function fetchSessions(manual)'); +const inflightDecl = /var sessionsInflight = null;/.exec(html); +const seqDecl = /var sessionsSeq = 0;/.exec(html); +if (!fetchSessionsSrc || !inflightDecl || !seqDecl) { + failures.push('session list: could not lift fetchSessions and the state it keeps'); +} else { + vm.runInContext(inflightDecl[0] + '\n' + seqDecl[0] + '\n' + fetchSessionsSrc, sandbox); + + // A fetch the section drives: every request gets a promise the section settles, and the + // signal is the real one, so an abort is an abort. + const requests = []; + sandbox.fetch = function (url, options) { + const entry = { + url: url, + signal: options && options.signal, + // The headers the page sent, so a section can ask what the request claimed about the + // list it already had. A validator is only ever read by the server, so the only way to + // see whether the page still holds the right one is to look at what it asked with. + headers: (options && options.headers) || {}, + settle: null, + fail: null + }; + entry.promise = new Promise((resolve, reject) => { + entry.settle = resolve; + entry.fail = reject; + }); + requests.push(entry); + return entry.promise; + }; + sandbox.AbortController = AbortController; + sandbox.state = { + sessions: [], children: [], sessionsEtag: '', sessionsEtagFor: '', sessionsLoaded: false + }; + sandbox.noteFresh = () => false; + sandbox.renderBars = () => {}; + sandbox.freshnessReason = () => 'unknown'; + sandbox.renderSessionSelect = () => {}; + sandbox.renderChildSelect = () => {}; + + const abortError = () => { + const error = new Error('The operation was aborted.'); + error.name = 'AbortError'; + return error; + }; + const answer = (entry, body, etag) => entry.settle({ + status: 200, + ok: true, + headers: { get: (name) => (name === 'etag' ? (etag || '') : '') }, + json: () => Promise.resolve(body) + }); + + // Three requests: the first is replaced by the second, the second is replaced by the + // third. The second's answer arrives last, and it is the older one. + const first = sandbox.fetchSessions(false); + const second = sandbox.fetchSessions(false); + check('the second request stops the first', requests[0].signal.aborted, true); + // The browser delivers an abort as a rejection, and that rejection is handled while the + // second request is already in flight -- which is exactly when the controller used to be + // cleared. + requests[0].fail(abortError()); + await first; + const third = sandbox.fetchSessions(false); + check('a replaced request does not take the controller of the request that replaced it', + requests[1].signal.aborted, true); + answer(requests[2], { sessions: [{ id: 'mvs_new_main' }], children: [{ id: 'mvs_new_run' }] }, '"older"'); + await third; + answer(requests[1], { sessions: [{ id: 'mvs_old_main' }], children: [{ id: 'mvs_old_run' }] }); + await second; + check('a late answer does not replace the newer list on screen', + sandbox.state.sessions.map((row) => row.id), ['mvs_new_main']); + check('and the run list is the newer one too', + sandbox.state.children.map((row) => row.id), ['mvs_new_run']); + + // A 304 that arrives late. This is the second case of the same rule, and it is a different + // case: a 200 carries a body, so a stale one has something visible to overwrite. A 304 + // carries a VALIDATOR instead, so a stale one has nothing to overwrite but the tag that + // describes the list on screen -- and that tag is what the next poll asks the server with. + // + // It used to write that tag before the sequence gate, so the list on screen and the tag that + // described it were two different things: the next poll asked the server about the old list, + // the server said "not modified", and the page kept the new one. + const late = sandbox.fetchSessions(false); + const lateReplacement = sandbox.fetchSessions(false); + answer(requests[4], { sessions: [{ id: 'mvs_newest_main' }], children: [] }, '"newest"'); + await lateReplacement; + check('the newer list is on screen', + sandbox.state.sessions.map((row) => row.id), ['mvs_newest_main']); + requests[3].settle({ + status: 304, + ok: false, + headers: { get: (name) => (name === 'etag' ? '"older"' : '') } + }); + await late; + check('a late 304 does not overwrite the validator the newer answer installed', + sandbox.state.sessionsEtag, '"newest"'); + check('and the list on screen is still the newer one', + sandbox.state.sessions.map((row) => row.id), ['mvs_newest_main']); + + // And the poll after it asks the server about the list that is on screen. + const next = sandbox.fetchSessions(false); + check('the next request carries the validator of the list on screen', + requests[5].headers['If-None-Match'], '"newest"'); + + // A manual refresh drops the validator on purpose: the reader asked for the list again, so + // the request must not claim to already have it. + sandbox.fetchSessions(true); + check('a manual refresh sends no validator', + requests[6].headers['If-None-Match'], undefined); +} + +// The trajectory poll is the second implementation of the same rule the session list follows, +// and it used to write the validator BEFORE its sequence gate. A 304 that arrived late then +// put its own older tag over the newer one, so the body on screen and the tag that described +// it were two different things: the next poll asked the server about the old body and kept the +// new one on screen. +// +// The counterexample, driven against the shipped function: the page holds A, two polls are in +// flight, the newer one answers first with B and B's tag, and the older one answers with a 304 +// carrying A's tag. The old tag must not survive that. +const fetchTrajectorySrc = fnSource('function fetchTrajectory(manual)'); +const sessionParamSrc = fnSource('function sessionParam()'); +const trajInflightDecl = /var inflight = null;/.exec(html); +const trajSeqDecl = /var refreshSeq = 0;/.exec(html); +if (!fetchTrajectorySrc || !sessionParamSrc || !trajInflightDecl || !trajSeqDecl) { + failures.push('trajectory poll: could not lift fetchTrajectory and the state it keeps'); +} else { + vm.runInContext( + trajInflightDecl[0] + '\n' + trajSeqDecl[0] + '\n' + sessionParamSrc + '\n' + fetchTrajectorySrc, + sandbox + ); + + const trajRequests = []; + sandbox.fetch = function (url, options) { + const entry = { + url: url, + signal: options && options.signal, + headers: (options && options.headers) || {}, + settle: null + }; + entry.promise = new Promise((resolve) => { entry.settle = resolve; }); + trajRequests.push(entry); + return entry.promise; + }; + sandbox.AbortController = AbortController; + sandbox.state = { + payload: null, generation: null, etag: '', etagFor: '', since: 0, sinceFor: '', + loading: false, followLatest: true, sessionId: '', hostSessionId: '' + }; + sandbox.renderLoading = () => {}; + sandbox.noteFresh = () => false; + sandbox.render = () => {}; + const applied = []; + sandbox.applyPayload = function (body) { applied.push(body); }; + + const trajAnswer = (entry, body, etag) => entry.settle({ + status: 200, + ok: true, + headers: { get: (name) => (name === 'etag' ? (etag || '') : '') }, + json: () => Promise.resolve(body) + }); + const trajNotModified = (entry, etag) => entry.settle({ + status: 304, + ok: false, + headers: { get: (name) => (name === 'etag' ? (etag || '') : '') } + }); + + const older = sandbox.fetchTrajectory(false); + const newer = sandbox.fetchTrajectory(false); + check('the second poll stops the first', trajRequests[0].signal.aborted, true); + trajAnswer(trajRequests[1], { session: { id: 'mvs_b' } }, '"tag-b"'); + await newer; + check('the newer body is the one on screen', applied.map((b) => b.session.id), ['mvs_b']); + check('and the validator describes it', sandbox.state.etag, '"tag-b"'); + + // The older answer arrives late and carries the older tag. + trajNotModified(trajRequests[0], '"tag-a"'); + await older; + check('a replaced trajectory request writes no validator', sandbox.state.etag, '"tag-b"'); + check('and it leaves the body on screen alone', + applied.map((b) => b.session.id), ['mvs_b']); + + // And the poll after it asks the server about the body that is on screen. + sandbox.fetchTrajectory(false); + check('the next poll carries the validator of the body on screen', + trajRequests[2].headers['If-None-Match'], '"tag-b"'); + + // The same rule for a late 200: a replaced request must not put its body on screen either. + const lateOld = sandbox.fetchTrajectory(false); + const lateNew = sandbox.fetchTrajectory(false); + trajAnswer(trajRequests[4], { session: { id: 'mvs_c' } }, '"tag-c"'); + await lateNew; + trajAnswer(trajRequests[3], { session: { id: 'mvs_a' } }, '"tag-a"'); + await lateOld; + check('a replaced trajectory request writes no body either', + applied.map((b) => b.session.id), ['mvs_b', 'mvs_c']); + check('and the validator is still the newer one', sandbox.state.etag, '"tag-c"'); +} + +// A record index is a position within ONE body, so a read that carries that index has to ask +// about the session that body was built for. The client already says so in its own cache key — +// `rawCacheKey` names the displayed session — but the request used to ask for `latest`, which +// is "whichever session is current now" and not necessarily the one on screen. The runtime can +// make another session current while the page still shows this one, and the page then showed +// that other session's words under this session's record. +const requestRawSrc = fnSource('function requestRaw(index, version)'); +const requestDetailSrc = fnSource('function requestDetail(index, version)'); +const displayedSessionSrc = fnSource('function displayedSessionParam()'); +const displayedGenerationSrc = fnSource('function displayedGeneration()'); +const recordVersionOfSrc = fnSource('function recordVersionOf(record)'); +const rawCacheKeySrc = fnSource('function rawCacheKey(index, version)'); +if (!requestRawSrc || !requestDetailSrc || !displayedSessionSrc || !displayedGenerationSrc + || !recordVersionOfSrc || !rawCacheKeySrc) { + failures.push('secondary reads: could not lift requestRaw, requestDetail and the session they ask for'); +} else { + vm.runInContext(displayedSessionSrc + '\n' + displayedGenerationSrc + '\n' + + recordVersionOfSrc + '\n' + + requestRawSrc + '\n' + requestDetailSrc + '\n' + rawCacheKeySrc, sandbox); + + const secondary = []; + sandbox.fetch = function (url) { + secondary.push(url); + return Promise.resolve({ + ok: true, + status: 200, + json: () => Promise.resolve({ raw: 'ALPHA BODY', index: 1, text: 'ALPHA BODY' }) + }); + }; + // The page is following the newest conversation, and it is showing A. The runtime may already + // have made B current — the page does not know that, and it must not care: A is on screen. + // + // `flat` is the body on screen's records. A read that names an index has to name the RECORD + // behind that index, and the record's version is what says which one it is: the same position + // in a newer body can be a different record entirely. + sandbox.state = { + payload: { session: { id: 'mvs_displayed', generation: null } }, + flat: [{ turn: {}, record: { index: 1, v: 'the-version-of-record-one' } }], + generation: null, etag: '"tag"', followLatest: true, sessionId: '', hostSessionId: '' + }; + await sandbox.requestRaw(1, 'the-version-of-record-one'); + await sandbox.requestDetail(1, 'the-version-of-record-one'); + check('a raw read asks for the session on screen', + /[?&]session=mvs_displayed(&|$)/.test(secondary[0]), true); + check('and it does not ask for whatever is current now', + /[?&]session=latest(&|$)/.test(secondary[0]), false); + check('a detail read asks for the session on screen', + /[?&]session=mvs_displayed(&|$)/.test(secondary[1]), true); + check('and it does not ask for whatever is current now', + /[?&]session=latest(&|$)/.test(secondary[1]), false); + check('a raw read names the version of the record it is asking for', + /[?&]v=the-version-of-record-one(&|$)/.test(secondary[0]), true); + check('and a detail read names it too', + /[?&]v=the-version-of-record-one(&|$)/.test(secondary[1]), true); + + // The slice on screen is not the generation the reader picked. A selection whose load failed + // leaves the previous body up, and a read that asks about the selection asks about a body + // that is not there — its answer is about records the reader cannot see. + sandbox.state = { + payload: { session: { id: 'mvs_displayed', generation: 2 } }, + flat: [{ turn: {}, record: { index: 1, v: 'the-version-of-record-one' } }], + generation: 7, etag: '"tag"', followLatest: true, sessionId: '', hostSessionId: '' + }; + await sandbox.requestRaw(1, 'the-version-of-record-one'); + await sandbox.requestDetail(1, 'the-version-of-record-one'); + check('a raw read asks for the slice on screen', + /[?&]generation=2(&|$)/.test(secondary[2]), true); + check('and not for the one the reader picked', + /[?&]generation=7(&|$)/.test(secondary[2]), false); + check('a detail read asks for the slice on screen too', + /[?&]generation=2(&|$)/.test(secondary[3]), true); + check('and not for the one the reader picked', + /[?&]generation=7(&|$)/.test(secondary[3]), false); + + // The browser's own raw cache is keyed the same way, and for the same reason: a record index + // is a position inside one body, and the body on screen is not always the body the reader last + // selected. A key built from the selection would serve the previous slice's message under the + // current slice's record — and the reader would be reading another generation's words. + check('the raw cache key names the slice on screen', + sandbox.rawCacheKey(1).includes('#2#'), true); + check('and not the one the reader picked', + sandbox.rawCacheKey(1).includes('#7#'), false); + check('and it still names the session on screen', + sandbox.rawCacheKey(1).startsWith('mvs_displayed#'), true); + + // With nothing on screen there is no session to name, and the read falls back to the session + // parameter rather than sending an empty one. + sandbox.state = { payload: null, generation: null, etag: '', followLatest: true, sessionId: '', hostSessionId: '' }; + await sandbox.requestRaw(1, 'the-version-of-record-one'); + check('with no body on screen the read falls back to the current session', + /[?&]session=latest(&|$)/.test(secondary[4]), true); +} + +/* A read that is retried must ask about the SAME record. + * + * The retry exists because the service let go of the session: it reloads the body and asks again. + * The body it reloads is a DIFFERENT one — the transcript moved while the tab was open — so a + * version read off the screen at that moment belongs to the NEW body's record at that index, and + * the retry is a different question wearing the same question's clothes. The service would answer + * it, because it is a perfectly good question about a perfectly good record: the wrong one. + * + * So the version travels with the RECORD the reader opened, and the reload does not change it. */ +{ + const fetchRawSrc = fnSource('function fetchRaw(record, retried)'); + const fetchDetailSrc = fnSource('function fetchDetail(record, retried)'); + const requestDetailSrc2 = fnSource('function requestDetail(index, version)'); + const isCountSrc = fnSource('function isCount(value)'); + if (!fetchRawSrc || !fetchDetailSrc || !requestRawSrc || !requestDetailSrc2 || !isCountSrc + || !recordVersionOfSrc) { + failures.push('secondary read retry: could not lift fetchRaw, fetchDetail and the requests they make'); + } else { + vm.runInContext('var rawCache = {};\nvar rawPending = {};\nvar detailCache = {};\nvar detailPending = {};\n' + + isCountSrc + '\n' + recordVersionOfSrc + '\n' + + requestRawSrc + '\n' + requestDetailSrc2 + '\n' + + fetchRawSrc + '\n' + fetchDetailSrc, sandbox); + + const retried = []; + let attempts = 0; + sandbox.fetch = function (url) { + retried.push(String(url)); + attempts += 1; + if (attempts === 1 || attempts === 3) { + return Promise.resolve({ ok: false, status: 503, + json: () => Promise.resolve({ error: 'trajectory_not_cached' }) }); + } + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ raw: 'ALPHA BODY', index: 1, text: 'ALPHA BODY', omitted: false }) }); + }; + // The reload is what brings the NEW body: the same index, another record. A version read off + // the screen after this point is the new record's version, and the retry would be asking + // about it. + sandbox.fetchTrajectory = function () { + sandbox.state.flat = [{ turn: {}, record: { index: 1, v: 'a-newer-version' } }]; + return Promise.resolve(); + }; + sandbox.state = { + payload: { session: { id: 'mvs_displayed', generation: null } }, + flat: [{ turn: {}, record: { index: 1, v: 'the-version-of-record-one' } }], + generation: null, etag: '"tag"', followLatest: true, sessionId: '', hostSessionId: '' + }; + const opened = { index: 1, v: 'the-version-of-record-one' }; + await sandbox.fetchRaw(opened); + await sandbox.fetchDetail(opened); + check('the first read names the version of the record the reader opened', + /[?&]v=the-version-of-record-one(&|$)/.test(retried[0] || ''), true); + check('a read that is retried asks about that same record', + /[?&]v=the-version-of-record-one(&|$)/.test(retried[1] || ''), true); + check('and not about the record that happens to be at that index after the reload', + /[?&]v=a-newer-version(&|$)/.test(retried[1] || ''), false); + check('and the same for the record\'s fields', + /[?&]v=the-version-of-record-one(&|$)/.test(retried[2] || ''), true); + check('a field read that is retried asks about that same record too', + /[?&]v=the-version-of-record-one(&|$)/.test(retried[3] || ''), true); + check('and not about the record that happens to be at that index after the reload', + /[?&]v=a-newer-version(&|$)/.test(retried[3] || ''), false); + + /* The retry must not JOIN another read. + * + * The reload brings a body in which the record at that position is a DIFFERENT one, and the + * panel that rebuilds on it asks for the record now on screen. The read of the record the + * reader opened is still in flight, and it has to ask about ITS OWN record: the two reads are + * at the same position, and the key that decides whose answer a caller gets is the only + * thing that can tell them apart. + * + * The key used to be built out of what is on screen — the validator, the slice, the position — + * so after the reload the retry of the old record and the new record's own read had the SAME + * key. The retry was then answered with the new record's body without asking anything: the + * reader opened ALPHA and the page handed back CHARLIE. The version belongs in the key, and + * it is the caller's version, not the screen's. */ + for (const [readName, what] of [['fetchRaw', 'a retried read'], + ['fetchDetail', 'a retried field read']]) { + vm.runInContext('rawCache = {}; rawPending = {}; detailCache = {}; detailPending = {};', sandbox); + const joined = []; + sandbox.state = { + payload: { session: { id: 'mvs_displayed', generation: null } }, + flat: [{ turn: {}, record: { index: 1, v: 'the-version-of-record-one' } }], + generation: null, etag: '"tag-old"', followLatest: true, sessionId: '', hostSessionId: '' + }; + sandbox.fetch = function (url) { + joined.push(String(url)); + // The first attempt is refused because the service let go of the session. + if (joined.length === 1) { + return Promise.resolve({ ok: false, status: 503, + json: () => Promise.resolve({ error: 'trajectory_not_cached' }) }); + } + // The old record's own request: the service refuses it, because the record at that + // position is not the one that was asked for. + if (/[?&]v=the-version-of-record-one(&|$)/.test(joined[joined.length - 1])) { + return Promise.resolve({ ok: false, status: 409, + json: () => Promise.resolve({ error: 'raw_stale' }) }); + } + // The record now on screen, which the rebuild of the panel asks for. + return Promise.resolve({ ok: true, status: 200, + json: () => Promise.resolve({ raw: 'CHARLIE BODY longer', index: 1, text: 'CHARLIE BODY longer' }) }); + }; + sandbox.fetchTrajectory = function () { + // The new body is on screen, with its own validator, and the panel rebuilds on it: that + // starts a read of the record now at that position. + // + // THE SAME KIND OF READ as the one under test. A field read that is retried has to be + // tested against a COMPETING FIELD READ: the two live in different pending maps, so a + // raw read competing with a field read would leave the field's own map empty, and the + // rule — a retry must not join another read — would never have been asked of it. + sandbox.state.etag = '"tag-new"'; + sandbox.state.flat = [{ turn: {}, record: { index: 1, v: 'the-version-of-record-two' } }]; + sandbox[readName]({ index: 1, v: 'the-version-of-record-two' }); + return Promise.resolve(); + }; + let refused = null; + let handed = null; + try { + handed = await sandbox[readName]({ index: 1, v: 'the-version-of-record-one' }); + } catch (err) { + refused = err && err.code ? err.code : String(err); + } + check(what + ' cannot join a newer record\'s pending read after reload', + refused, 'raw_stale'); + check(what + ' is not answered with the other record\'s body', handed, null); + check(what + ' asked about the record the reader opened, on a request of its own', + joined.filter((u) => /[?&]v=the-version-of-record-one(&|$)/.test(u)).length, 2); + } + } +} + console.log('[2] pure helpers: ' + pass + ' passed, ' + failures.length + ' failed'); if (failures.length) { console.log('FAILURES:\n - ' + failures.join('\n - ')); diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-artifact-root.py b/plugins/avatasia/mmc-trajectory/tests/mutate-artifact-root.py new file mode 100644 index 00000000..1208d542 --- /dev/null +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-artifact-root.py @@ -0,0 +1,166 @@ +"""Mutation controls for verify-artifact-root.mjs. + +Each entry below puts one of the rule's owners back the way it was before the rule existed, and +requires the guard to report it BY NAME before the file is restored byte for byte. A mutation +reported as survived means the guard it exercised is not testing anything, and the fix is the +guard, never this list. + +Five owners are covered, and they are not the same kind of thing: + + - the runtime refusing a data dir inside the package at all; + - the refusal reaching a directory UNDER the package, not just the package itself; + - the refusal not being so broad that it turns into a refusal of a sibling whose name merely + starts with the package's name; + - the package ignoring the directory its artifacts would land in; + - the shipped harnesses not handing the package over as their data dir. + +Nothing here needs a network, a service, or a specific machine's paths. +""" +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +PROBE = HERE / "verify-artifact-root.mjs" + +SERVER = "miniapp/node/server.mjs" +GITIGNORE = ".gitignore" +LAYERS = "tests/verify-session-layers.mjs" + +MUTATIONS = [ + # ---- the runtime stops refusing ------------------------------------------------------ + ( + "the runtime stops refusing a data dir inside the package", + SERVER, + " assertDataDirOutsidePlugin(context);\n", + "", + "the runtime refuses a data dir that is the plugin package", + ), + # ---- the refusal has to reach a directory under it, not only the package itself ------- + ( + "the refusal only catches the package itself, not a directory under it", + SERVER, + " if (isUnderDirectory(pluginRoot, dataDir)) {", + " if (resolve(dataDir) === resolve(pluginRoot)) {", + "the runtime refuses a data dir under the plugin package", + ), + # ---- and it must not be so broad that it refuses everything -------------------------- + ( + "the refusal is a prefix test, so a sibling of the package is refused too", + SERVER, + " if (isUnderDirectory(pluginRoot, dataDir)) {", + " if (resolve(dataDir).startsWith(resolve(pluginRoot))) {", + "the runtime accepts a sibling of the plugin package whose name starts with its name", + ), + # ---- the package ignores where its artifacts would land ------------------------------ + ( + "the package ships without ignoring the directory its artifacts would land in", + GITIGNORE, + "\ntrajectory-artifacts/", + "", + "the package ignores the directory its artifacts would land in", + ), + # ---- and no shipped harness hands the package over ----------------------------------- + ( + "a shipped harness hands the package to the runtime as its data dir", + LAYERS, + " const port = 19400 + Math.floor(Math.random() * 400) + attempt;\n" + " try {\n" + " instance = await start({\n" + " pluginId: 'mmc-trajectory',\n" + " pluginRoot: PLUGIN_ROOT,\n" + " dataDir: DATA_DIR,", + " const port = 19400 + Math.floor(Math.random() * 400) + attempt;\n" + " try {\n" + " instance = await start({\n" + " pluginId: 'mmc-trajectory',\n" + " pluginRoot: PLUGIN_ROOT,\n" + " dataDir: PLUGIN_ROOT,", + "no shipped harness hands the package to the runtime as its data dir", + ), +] + + +def read_exact(path): + """Normalise to LF for matching, and report the file's own newline so it can be put back.""" + data = path.read_bytes() + text = data.decode("utf-8").replace("\r\n", "\n") + return text, ("\r\n" if b"\r\n" in data else "\n") + + +def write_exact(path, text, newline): + if newline == "\r\n": + text = text.replace("\n", "\r\n") + path.write_bytes(text.encode("utf-8")) + + +def run_probe(): + proc = subprocess.run( + ["node", str(PROBE)], + cwd=str(ROOT), + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + return proc.returncode, (proc.stdout or "") + (proc.stderr or "") + + +def main(): + code, output = run_probe() + if code != 0: + print("baseline probe does not pass; fix that first") + print(output) + return 1 + print("baseline clean") + + originals = {rel: (ROOT / rel).read_bytes() for rel in (SERVER, GITIGNORE, LAYERS)} + + missed = [] + unnamed = [] + for label, rel, old, new, expected in MUTATIONS: + target = ROOT / rel + text, newline = read_exact(target) + if text.count(old) != 1: + print(f"SKIP {label}: the anchor resolves {text.count(old)} times, expected exactly 1") + missed.append(label) + continue + if old == new: + print(f"SKIP {label}: the mutation changes nothing") + missed.append(label) + continue + write_exact(target, text.replace(old, new, 1), newline) + try: + code, output = run_probe() + finally: + write_exact(target, text, newline) + if code == 0: + print(f"MISSED {label}: the probe still passed") + missed.append(label) + continue + if expected not in output: + print(f"UNNAMED {label}: the probe failed without naming 「{expected}」") + unnamed.append(label) + continue + first = next((line.strip() for line in output.splitlines() if line.strip().startswith("- ")), None) + print(f"caught {label}\n -> {first}") + + for rel, original in originals.items(): + if (ROOT / rel).read_bytes() != original: + print(f"NOT RESTORED {rel}") + return 1 + print("files restored byte-for-byte") + + if missed or unnamed: + if missed: + print(f"\n{len(missed)} mutation(s) got through") + if unnamed: + print(f"\n{len(unnamed)} mutation(s) failed without naming a rule") + return 1 + print(f"\nall {len(MUTATIONS)} mutations caught, each by name") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-compaction-turn.py b/plugins/avatasia/mmc-trajectory/tests/mutate-compaction-turn.py index 2253f34b..3b314ddc 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-compaction-turn.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-compaction-turn.py @@ -60,10 +60,15 @@ " var opened = false", ), ( + # Re-anchored: the routing moved out of renderLedger() and into fillGroupRows(), which + # is the one place a group's rows are built. The rule did not change, only where it + # lives. The decision and the branch that uses it are two lines apart here, so the + # branch is what gets broken — and the guard that owns the rule now reads the branch + # rather than the word. "client: the ledger stops routing checkpoints to the band", CLIENT, - " if (group.entries[j].record.kind === 'compaction') {", - " if (false) {", + " if (isBand) {\n", + " if (false) {\n", ), ] diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-css.py b/plugins/avatasia/mmc-trajectory/tests/mutate-css.py index f1b02ed6..da9d0e56 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-css.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-css.py @@ -19,7 +19,13 @@ Each mutation must be caught *by name*. A probe that dies on an exception names nothing, and a control that reports that as "caught" is reporting a crash as if it were evidence. + +Every case names the rule it is about in its `expect` field, and a case counts as caught only +when the probe reported THAT rule: a probe that went red on some other assertion has not shown +that this guard owns this mutation. """ +import json +import re import subprocess import sys from pathlib import Path @@ -48,17 +54,20 @@ } .kpi-section {""", + "every class selector in the stylesheet has a consumer outside it", ), ( "the turn header loses its meta line and the rule stays behind", TURN_META_CONSUMER, "var meta = el('span');", + "every class selector in the stylesheet has a consumer outside it", ), ( "a custom property is declared in a theme block and never read", LIGHT_THEME_TAIL, """ --mcode-danger: #f73646; --mcode-never-read: #123456;""", + "--mcode-never-read is read somewhere", ), ] @@ -88,13 +97,43 @@ def run_probe(): return proc.returncode, (proc.stdout or "") + (proc.stderr or "") -def named_failure(output): - """The first rule the probe named, or None when it failed without naming one.""" - return next( - (line.strip()[len("FAIL"):].strip() for line in output.splitlines() - if line.strip().startswith("FAIL")), - None, - ) +def rule_name(text): + """The name of a rule a probe reported, without the evidence a comparison carries. + + A probe reports an assertion as `` and a comparison as + `: got , want `. The values are the evidence, not the rule, so they are + cut off — and the cut is made at a `: got ` whose tail really is `, want `, so a + name that itself contains those words is not truncated. + """ + for match in reversed(list(re.finditer(re.escape(": got "), text))): + head = text[:match.start()].strip() + tail = text[match.end():] + cut = tail.rfind(", want ") + if cut < 0: + continue + try: + json.loads(tail[:cut]) + json.loads(tail[cut + len(", want "):]) + except ValueError: + continue + return head + return text + + +def reported_rules(output): + """The rules a probe named as broken. + + Two formats, as in `mutate-search-endpoint.py`: ` - ` and `FAIL `. A case is + only caught when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(rule_name(text[2:].strip())) + elif text.startswith("FAIL "): + names.append(rule_name(text[5:].strip())) + return names def main(): @@ -108,7 +147,7 @@ def main(): original = CLIENT.read_bytes() missed = [] unnamed = [] - for label, old, new in MUTATIONS: + for label, old, new, expect in MUTATIONS: text, newline = read_exact(CLIENT) if old not in text: print(f"SKIP {label}: anchor not found") @@ -131,12 +170,19 @@ def main(): print(f"MISSED {label}: the probe still passed") missed.append(label) continue - name = named_failure(output) - if name is None: + names = reported_rules(output) + if not names: print(f"UNNAMED {label}: the probe failed without naming a rule") unnamed.append(label) - else: - print(f"caught {label}\n -> {name}") + continue + # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") if CLIENT.read_bytes() != original: print(f"NOT RESTORED {CLIENT}") diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-dead-code.py b/plugins/avatasia/mmc-trajectory/tests/mutate-dead-code.py index 1173dcd7..a58c0b0b 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-dead-code.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-dead-code.py @@ -21,7 +21,13 @@ Each must be caught *by name*. A probe that dies on an exception names nothing, and a control that reports that as caught is reporting a crash as if it were evidence. + +Every case names the rule it is about in its `expect` field, and a case counts as caught only +when the probe reported THAT rule: a probe that went red on some other assertion has not shown +that this guard owns this mutation. """ +import json +import re import subprocess import sys from pathlib import Path @@ -47,16 +53,19 @@ } function axisIcon() {""", + "every function the client declares is used by something", ), ( "a caller is deleted and the helper is left behind", AXIS_ICON_CALLER, " axis.appendChild(document.createTextNode(''));", + "every function the client declares is used by something", ), ( "the only reference to a function is left in a comment", AXIS_ICON_CALLER, " // was: axis.appendChild(axisIcon())\n axis.appendChild(document.createTextNode(''));", + "every function the client declares is used by something", ), ] @@ -86,13 +95,43 @@ def run_probe(): return proc.returncode, (proc.stdout or "") + (proc.stderr or "") -def named_failure(output): - """The first rule the probe named, or None when it failed without naming one.""" - return next( - (line.strip()[len("FAIL"):].strip() for line in output.splitlines() - if line.strip().startswith("FAIL")), - None, - ) +def rule_name(text): + """The name of a rule a probe reported, without the evidence a comparison carries. + + A probe reports an assertion as `` and a comparison as + `: got , want `. The values are the evidence, not the rule, so they are + cut off — and the cut is made at a `: got ` whose tail really is `, want `, so a + name that itself contains those words is not truncated. + """ + for match in reversed(list(re.finditer(re.escape(": got "), text))): + head = text[:match.start()].strip() + tail = text[match.end():] + cut = tail.rfind(", want ") + if cut < 0: + continue + try: + json.loads(tail[:cut]) + json.loads(tail[cut + len(", want "):]) + except ValueError: + continue + return head + return text + + +def reported_rules(output): + """The rules a probe named as broken. + + Two formats, as in `mutate-search-endpoint.py`: ` - ` and `FAIL `. A case is + only caught when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(rule_name(text[2:].strip())) + elif text.startswith("FAIL "): + names.append(rule_name(text[5:].strip())) + return names def main(): @@ -106,7 +145,7 @@ def main(): original = CLIENT.read_bytes() missed = [] unnamed = [] - for label, old, new in MUTATIONS: + for label, old, new, expect in MUTATIONS: text, newline = read_exact(CLIENT) if old not in text: print(f"SKIP {label}: anchor not found") @@ -129,12 +168,19 @@ def main(): print(f"MISSED {label}: the probe still passed") missed.append(label) continue - name = named_failure(output) - if name is None: + names = reported_rules(output) + if not names: print(f"UNNAMED {label}: the probe failed without naming a rule") unnamed.append(label) - else: - print(f"caught {label}\n -> {name}") + continue + # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") if CLIENT.read_bytes() != original: print(f"NOT RESTORED {CLIENT}") diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-file-index-rows.py b/plugins/avatasia/mmc-trajectory/tests/mutate-file-index-rows.py index e8714df2..d6463110 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-file-index-rows.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-file-index-rows.py @@ -24,7 +24,13 @@ Each mutation must be caught *by name*. A probe that dies on an exception names nothing, and a control that reports that as caught is reporting a crash as if it were evidence. + +Every case names the rule it is about in its `expect` field, and a case counts as caught only +when the probe reported THAT rule: a probe that went red on some other assertion has not shown +that this guard owns this mutation. """ +import json +import re import subprocess import sys from pathlib import Path @@ -56,21 +62,25 @@ "a record with no locator is repaired to the first file instead of -1", ROW_UNLOCATED, UNLOCATED_REPAIRED, + "a record with no locator says file -1", ), ( "a locator whose file moved is filed under the first file instead of -1", ROW, MOVED_REPAIRED, + "a locator whose file is not in the table says file -1, not file 0", ), ( "two generations holding the same file name are matched by basename", FULL_PATH, BASENAME, + "each record names the file its locator names", ), ( "the byte range is copied onto the row again, so it rides along on every segment", ROW_UNLOCATED, BYTES_BACK, + "the byte range the locator carries is dropped, not copied onto the row", ), ] @@ -100,13 +110,43 @@ def run_probe(): return proc.returncode, (proc.stdout or "") + (proc.stderr or "") -def named_failure(output): - """The first rule the probe named, or None when it failed without naming one.""" - return next( - (line.strip()[len("FAIL"):].strip() for line in output.splitlines() - if line.strip().startswith("FAIL")), - None, - ) +def rule_name(text): + """The name of a rule a probe reported, without the evidence a comparison carries. + + A probe reports an assertion as `` and a comparison as + `: got , want `. The values are the evidence, not the rule, so they are + cut off — and the cut is made at a `: got ` whose tail really is `, want `, so a + name that itself contains those words is not truncated. + """ + for match in reversed(list(re.finditer(re.escape(": got "), text))): + head = text[:match.start()].strip() + tail = text[match.end():] + cut = tail.rfind(", want ") + if cut < 0: + continue + try: + json.loads(tail[:cut]) + json.loads(tail[cut + len(", want "):]) + except ValueError: + continue + return head + return text + + +def reported_rules(output): + """The rules a probe named as broken. + + Two formats, as in `mutate-search-endpoint.py`: ` - ` and `FAIL `. A case is + only caught when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(rule_name(text[2:].strip())) + elif text.startswith("FAIL "): + names.append(rule_name(text[5:].strip())) + return names def main(): @@ -120,7 +160,7 @@ def main(): original = MODULE.read_bytes() missed = [] unnamed = [] - for label, old, new in MUTATIONS: + for label, old, new, expect in MUTATIONS: text, newline = read_exact(MODULE) if old not in text: print(f"SKIP {label}: anchor not found") @@ -143,12 +183,19 @@ def main(): print(f"MISSED {label}: the probe still passed") missed.append(label) continue - name = named_failure(output) - if name is None: + names = reported_rules(output) + if not names: print(f"UNNAMED {label}: the probe failed without naming a rule") unnamed.append(label) - else: - print(f"caught {label}\n -> {name.splitlines()[0]}") + continue + # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") if MODULE.read_bytes() != original: print(f"NOT RESTORED {MODULE}") diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-freshness.py b/plugins/avatasia/mmc-trajectory/tests/mutate-freshness.py index 666f4d15..168ca652 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-freshness.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-freshness.py @@ -8,7 +8,13 @@ The last case is the other direction: an edit that only ADDS A COMMENT must leave the probe green. A guard that cannot pass is as broken as one that cannot fail, and that is the only way to prove this probe is reading behaviour rather than prose. + +Every case names the rule it is about in its `expect` field, and a case counts as caught only +when the probe reported THAT rule: a probe that went red on some other assertion has not shown +that this guard owns this mutation. """ +import json +import re import subprocess import sys from pathlib import Path @@ -24,6 +30,7 @@ "a 304 is taken as nothing having happened, so the warning outlives the fault", " var cleared = noteFresh('trajectory', true, '');\n if (state.loading) {", " var cleared = false;\n if (state.loading) {", + "a 304 takes the warning down — this is the defect the guard exists for", ), ( # The other half of the same layer: the list failure used to be written to a flag that @@ -31,24 +38,28 @@ "a failed session list is recorded and never shown", " noteFresh('sessions', false, freshnessReason(error));\n renderBars();", " renderBars();", + "a failed list is visible at all — it used to write a flag nothing read", ), # ---- the count ------------------------------------------------------------------------ ( "recovering does not reset the count, so a past outage is remembered forever", " if (ok) {\n slot.streak = 0;\n slot.reason = '';", " if (ok) {\n slot.reason = '';", + "and the streak is back to zero", ), # ---- which kind of failure ------------------------------------------------------------- ( "a dropped connection is reported as a bad answer, which sends the reader the wrong way", " if (error && typeof error.status === 'number') return 'bad_response';\n return 'offline';", " return 'bad_response';", + "a dropped connection says so", ), # ---- the cost of the fix --------------------------------------------------------------- ( "every 304 re-renders, trading a stuck warning for a ledger rebuilt every 8 seconds", " if (cleared) render();", " render();", + "five more 304s on a quiet session render nothing", ), ] @@ -67,6 +78,11 @@ def write_exact(path, text, newline): def run_probe(): + """The probe's verdict, stdout and stderr together. + + This probe writes its failures to stderr and its summary to stdout, so both are captured: + a name that arrived on stderr is a name the guard reported. + """ proc = subprocess.run( ["node", str(PROBE)], cwd=str(ROOT), @@ -78,6 +94,45 @@ def run_probe(): return proc.returncode, (proc.stdout or "") + (proc.stderr or "") +def rule_name(text): + """The name of a rule a probe reported, without the evidence a comparison carries. + + A probe reports an assertion as `` and a comparison as + `: got , want `. The values are the evidence, not the rule, so they are + cut off — and the cut is made at a `: got ` whose tail really is `, want `, so a + name that itself contains those words is not truncated. + """ + for match in reversed(list(re.finditer(re.escape(": got "), text))): + head = text[:match.start()].strip() + tail = text[match.end():] + cut = tail.rfind(", want ") + if cut < 0: + continue + try: + json.loads(tail[:cut]) + json.loads(tail[cut + len(", want "):]) + except ValueError: + continue + return head + return text + + +def reported_rules(output): + """The rules a probe named as broken. + + Two formats, as in `mutate-search-endpoint.py`: ` - ` and `FAIL `. A case is + only caught when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(rule_name(text[2:].strip())) + elif text.startswith("FAIL "): + names.append(rule_name(text[5:].strip())) + return names + + def main(): code, output = run_probe() if code != 0: @@ -89,7 +144,7 @@ def main(): original = HTML.read_bytes() missed = [] unnamed = [] - for label, old, new in MUTATIONS: + for label, old, new, expect in MUTATIONS: text, newline = read_exact(HTML) if old not in text: print(f"SKIP {label}: anchor not found") @@ -108,12 +163,19 @@ def main(): print(f"MISSED {label}: the probe still passed") missed.append(label) continue - first = next((line.strip() for line in output.splitlines() if line.strip().startswith("- ")), None) - if first is None: + names = reported_rules(output) + if not names: print(f"UNNAMED {label}: the probe failed without naming a rule") unnamed.append(label) - else: - print(f"caught {label}\n -> {first}") + continue + # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") # ---- the other direction --------------------------------------------------------------- # A note in this file must not change a single verdict. If adding prose can turn the probe diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-handoff-identity.py b/plugins/avatasia/mmc-trajectory/tests/mutate-handoff-identity.py new file mode 100644 index 00000000..0daba4f8 --- /dev/null +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-handoff-identity.py @@ -0,0 +1,175 @@ +"""Mutation control for the session the handoff names. + +The handoff reads two things asynchronously — the record's untouched message and its four bulk +fields — and the page can be showing another conversation by the time they answer. The report it +hands to the Host NAMES the session, so the session has to be the one the selected record came +from, captured before the reads and not looked up after them. + +Reading it after them is what the first mutation below does, and it does it with getters rather +than with a moved line: the object handed to the markdown is captured, but its fields are read +when the markdown is built — which is after the reads have answered, and by then the page may be +showing somebody else's conversation. The getter is not how anyone writes code; it is how this +one defect is reproduced in a single edit, so the control cannot be satisfied by an +implementation that merely happens to look right. + +The second mutation is the blunter half: no session at all. + +Both have to be caught by the same named assertion in client-check.mjs, which runs the shipped +`handoffSelected` over a deferred raw read and moves the page while that read is in flight. + +Every case names the rule it is about in its `expect` field, and a case counts as caught only +when the probe reported THAT rule: a probe that went red on some other assertion has not shown +that this guard owns this mutation. +""" +import json +import re +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +CLIENT = ROOT / "miniapp" / "client" / "index.html" +PROBE = HERE / "client-check.mjs" + +MUTATIONS = [ + ( + "the handoff reads the session off the page after its reads, so a navigation names the wrong conversation", + " var session = state.payload ? state.payload.session : null;", + " var session = { get id() { return state.payload.session.id; }, get label() { return state.payload.session.label; } };", + 'the handoff names the session the selected record came from', + ), + ( + "the handoff names no conversation at all", + " var session = state.payload ? state.payload.session : null;", + " var session = null;", + 'the handoff names the session the selected record came from', + ), +] + + +def read_exact(path): + """Normalise to LF for matching, and report the file's own newline so it can be put back.""" + data = path.read_bytes() + text = data.decode("utf-8").replace("\r\n", "\n") + return text, ("\r\n" if b"\r\n" in data else "\n") + + +def write_exact(path, text, newline): + if newline == "\r\n": + text = text.replace("\n", "\r\n") + path.write_bytes(text.encode("utf-8")) + + +def run_probe(): + proc = subprocess.run( + ["node", str(PROBE)], + cwd=str(ROOT), + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + return proc.returncode, (proc.stdout or "") + (proc.stderr or "") + + +def rule_name(text): + """The name of a rule a probe reported, without the evidence a comparison carries. + + A probe reports an assertion as `` and a comparison as + `: got , want `. The values are the evidence, not the rule, so they are + cut off — and the cut is made at a `: got ` whose tail really is `, want `, so a + name that itself contains those words is not truncated. + """ + for match in reversed(list(re.finditer(re.escape(": got "), text))): + head = text[:match.start()].strip() + tail = text[match.end():] + cut = tail.rfind(", want ") + if cut < 0: + continue + try: + json.loads(tail[:cut]) + json.loads(tail[cut + len(", want "):]) + except ValueError: + continue + return head + return text + + +def reported_rules(output): + """The rules a probe named as broken. + + The probe prints `FAILURES:` and one ` - ` line per failure, so a case is only caught + when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(rule_name(text[2:].strip())) + elif text.startswith("FAIL "): + names.append(rule_name(text[5:].strip())) + return names + + +def main(): + code, output = run_probe() + if code != 0: + print("baseline probe does not pass; fix that first") + print(output) + return 1 + print("baseline clean") + + original = CLIENT.read_bytes() + missed = [] + unnamed = [] + for label, old, new, expect in MUTATIONS: + text, newline = read_exact(CLIENT) + if old not in text: + print(f"SKIP {label}: anchor not found") + missed.append(label) + continue + if old == new: + print(f"SKIP {label}: the mutation changes nothing") + missed.append(label) + continue + write_exact(CLIENT, text.replace(old, new, 1), newline) + try: + code, output = run_probe() + finally: + CLIENT.write_bytes(original) + if code == 0: + print(f"MISSED {label}: the probe still passed") + missed.append(label) + continue + names = reported_rules(output) + if not names: + print(f"UNNAMED {label}: the probe failed without naming a rule") + unnamed.append(label) + continue + # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") + + if CLIENT.read_bytes() != original: + print(f"NOT RESTORED {CLIENT}") + return 1 + print("file restored byte-for-byte") + + if missed or unnamed: + if missed: + print(f"\n{len(missed)} mutation(s) got through") + if unnamed: + print(f"{len(unnamed)} mutation(s) failed without naming a rule") + return 1 + print(f"\nall {len(MUTATIONS)} mutations caught, each by name") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-incremental.py b/plugins/avatasia/mmc-trajectory/tests/mutate-incremental.py index 6fe9d36b..9a4f0e02 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-incremental.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-incremental.py @@ -60,6 +60,22 @@ "the cursor after a delta is the server upTo, not a second walk", "state.since = mergeable ? body.delta.upTo : payloadCursor(payload);", "state.since = payloadCursor(payload);"), + # The set the tail is about, from the page's side. A record rewritten in place keeps its index + # and its id, so nothing else in the protocol can see that the page's copy is not the + # service's — the tail comes back empty and the page keeps the old text under the new body's + # validator. The service's half of this is in SERVER_CASES, because it edits server.mjs; a + # case that edits one file and sits in the other's list is a case whose anchor never resolves. + ("a tail is asked for without saying which set the page holds", + "a tail is asked for with the cursor and the set the page holds", + " + '&n=' + held.count\n", ""), + ("the page treats every failure as a refusal about its copy", + "a refusal about the page's copy is recognised as one, and only as one", + " return error.code === 'trajectory_version_mismatch'\n" + " || error.code === 'trajectory_fingerprint_required';\n", + " return Boolean(error);\n"), + ("the page asks for the whole body for something that is not a refusal", + "a tail the service will not answer leaves the page asking for the whole body", + " if (isStaleCopyRefusal(error)) {\n", " if (false) {\n"), ] SERVER_CASES = [ @@ -115,6 +131,17 @@ ("a whole turn ships and the count inherits the whole turn instead of the additions", "while still counting only what is new", " records += fresh;", " records += all.length;"), + + # The set the tail is about. A record rewritten in place keeps its index and its id, so + # nothing else in the protocol can see that the page's copy is not the service's: the tail + # comes back empty, and the page keeps the old text under the new body's validator. + ("a tail is answered to a page whose copy is not the one the service has", + "a content-only rewrite at the cursor cannot be acknowledged without replacing the displayed record", + "export function deltaRefusal(payload, count, askedFingerprint) {\n", + "export function deltaRefusal(payload, count, askedFingerprint) {\n if (payload) return null;\n"), + ("the refusal is computed and then ignored", + "the service refuses a tail whose set the page did not state, and says so", + " if (refusal) {\n", " if (false) {\n"), ] # The page side. A delta replayed against a copy that already has a record must be allowed to @@ -133,20 +160,31 @@ def run_probe(): ["node", str(PROBE)], cwd=str(ROOT), capture_output=True, text=True, encoding="utf-8", errors="replace", ) + lines = proc.stdout.splitlines() fired = [] - for line in proc.stdout.splitlines(): + for line in lines: if line.startswith("FAIL "): fired.append(line[5:].strip()) - # A crash is the probe failing to run, which the exit code says. It is *not* "no assertion - # fired" — this guard prints its FAIL lines only when there are any, so a passing probe - # prints none, and conflating the two turns every surviving mutation into a fake crash. - return proc.returncode == 0, fired, proc.stdout + proc.stderr + # Two facts, and they are not the same one. + # + # The exit code says whether the guard PASSED — and a guard that fails an assertion on purpose + # exits non-zero, so a non-zero code is what a CATCH looks like. Reading it as "the probe + # crashed" turns every catch into a crash, which is a control that reports nothing but crashes. + # + # Whether the probe REPORTED is the other fact. This guard prints its summary line whether it + # passed or failed, and a probe that died before its reporting block printed neither the + # summary nor any FAIL line. A run that produced no output contains no name, so a crash cannot + # be read as a catch. + # + # The verdict is the name. The exit code only says whether the baseline was clean. + reported = any("checks passed" in line for line in lines) + return proc.returncode, reported, fired, proc.stdout + proc.stderr def main(): originals = {CLIENT: CLIENT.read_bytes(), SERVER: SERVER.read_bytes()} - ok, fired, output = run_probe() - if not ok or fired: + code, reported, fired, output = run_probe() + if code != 0 or not reported or fired: print("baseline verify-incremental.mjs did not pass; refusing to run mutations") print(output) return 1 @@ -165,15 +203,18 @@ def main(): target.write_bytes(text.replace(old, new, 1).encode("utf-8")) try: - passed, fired, _ = run_probe() + code, reported, fired, _ = run_probe() finally: target.write_bytes(originals[target]) - if not ok: - failures.append("%s: CRASHED — the probe could not run; that is not a catch" % name) + # A probe that did not report died before it could name anything, and a run that produced + # no output contains no name — so neither of those is a catch. A probe that reported and + # named nothing failed nothing, which is a different thing: the rule still holds. + if not reported: + failures.append("%s: CRASHED — the probe did not report; that is not a catch" % name) continue if not fired: - failures.append("%s: NOT CAUGHT — probe still passed" % name) + failures.append("%s: NOT CAUGHT — the probe reported and passed anyway" % name) continue if expected not in fired: failures.append("%s: caught by %s, expected %r" % (name, fired, expected)) diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-inspector-dock.py b/plugins/avatasia/mmc-trajectory/tests/mutate-inspector-dock.py index 18cde867..9c320401 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-inspector-dock.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-inspector-dock.py @@ -133,6 +133,30 @@ " Math.round(window.innerHeight * INSPECTOR_LEDGER_SHARE)\n" " );\n" " var panelMax = Math.max(INSPECTOR_MIN_DOCK_H, inspectorUnderPins() - ledgerReserve);") +# The comment above the dock's fallback declaration, and the same comment without the constant's +# name in it. The block names the constant twice, so a mutation that dropped one of the two +# mentions would leave the other and the guard would still be right -- the anchor has to be the +# whole block. +DOCK_FALLBACK_COMMENT = ( + " /* The docked height, as the share of the window that INSPECTOR_DOCK_SHARE declares. It\n" + " was 50vh, then a quarter, and the unit was never the problem — the unit being frozen\n" + " was. The reader cannot make it smaller, and a cap the reader cannot move is a number\n" + " nobody chose. applyInspectorDock() therefore writes this in px from window.innerHeight\n" + " and a drag overwrites it, so the declaration below is only what shows before the first\n" + " measurement. It stays a vh for the same reason as everywhere else here: 100vh is not\n" + " the reader's height inside a host iframe, and every offset on this page is measured\n" + " for that reason. The declared vh must track INSPECTOR_DOCK_SHARE — the fallback and\n" + " the measurement are the same promise written twice, so a guard holds them together. */\n") +DOCK_FALLBACK_COMMENT_WITHOUT_NAME = ( + " /* The docked height, as the share of the window that the client declares. It\n" + " was 50vh, then a quarter, and the unit was never the problem — the unit being frozen\n" + " was. The reader cannot make it smaller, and a cap the reader cannot move is a number\n" + " nobody chose. applyInspectorDock() therefore writes this in px from window.innerHeight\n" + " and a drag overwrites it, so the declaration below is only what shows before the first\n" + " measurement. It stays a vh for the same reason as everywhere else here: 100vh is not\n" + " the reader's height inside a host iframe, and every offset on this page is measured\n" + " for that reason. The declared vh must track that share — the fallback and\n" + " the measurement are the same promise written twice, so a guard holds them together. */\n") MUTATIONS = [ # ---- the CSS, in the media query that makes the feature exist ---- @@ -445,6 +469,44 @@ " --inspector-dock-h: 25vh;\n", "the stylesheet fallback agrees with the measured share"), + # ---- the comments that state a share name the constant that holds it ---- + # + # Three comments say what share of the window the dock and the ledger take, and each of them + # names the constant rather than spelling the value out. The name is what a reader follows to + # the declaration, and it is what a rename has to move with it. + # + # The three mutations below each drop ONE name out of one comment and leave everything else + # alone — the comment still reads, the file still parses, and nothing about the behaviour + # changes. That is the point: the rule is about the comment, so only the comment moves. + + # The comment above the fallback declaration names the constant twice, so the anchor is the + # whole block: replacing one of the two mentions would leave the other, and the guard would + # still be right. + ("the comment above the dock fallback stops naming the constant", + DOCK_FALLBACK_COMMENT, + DOCK_FALLBACK_COMMENT_WITHOUT_NAME, + "the comment that states the dock default height names the constant that holds it"), + + # The comment above the docked rule names the constant twice as well, so the anchor spans both + # mentions: dropping one of them would leave the other, and the guard would still be right. + ("the comment above the docked rule stops naming the constant", + " /* Docked at INSPECTOR_DOCK_SHARE by default, and the reader can drag it. The content\n" + " scrolls inside the panel instead of the panel growing: a growing panel pushes its own\n" + " handle out of the viewport, which is the one thing a docked panel cannot do. The number\n" + " itself is written by applyInspectorDock() — INSPECTOR_DOCK_SHARE of the measured window,\n", + " /* Docked at the share the client declares by default, and the reader can drag it. The\n" + " content scrolls inside the panel instead of the panel growing: a growing panel pushes\n" + " its own handle out of the viewport, which is the one thing a docked panel cannot do. The\n" + " number itself is written by applyInspectorDock() — that share of the measured window,\n", + "the comment that states the docked rule names the constant it applies to"), + + ("the comment above the panel rule stops naming the ledger's constant", + " because that is already `innerHeight - pinned strips - the ledger's reserved share`\n" + " (INSPECTOR_LEDGER_SHARE): the panel can be neither taller than the window nor able to\n", + " because that is already `innerHeight - pinned strips - the ledger's reserved share`:\n" + " the panel can be neither taller than the window nor able to\n", + "the comment that states the ledger reserve names the constant that holds it"), + # The single writer is undone: one caller computes the share again and the other keeps calling # the helper. Both paths still produce a number, so nothing looks wrong until the two numbers # are different — which is the entire reason the helper exists. diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-intrinsic-size.py b/plugins/avatasia/mmc-trajectory/tests/mutate-intrinsic-size.py index 68c523a4..514863b3 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-intrinsic-size.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-intrinsic-size.py @@ -79,14 +79,14 @@ # Every group after the calibration stops being sized, which is the same collapse applied to # 127 of 152 groups instead of 25 — and the count of sized groups is what the measurement reads. ("later groups are left unsized", - [(" applyIntrinsicSize(section, group.entries.length, bands, build.heights);\n", "")], + [(" var predicted = applyIntrinsicSize(section, group.entries.length - bands, bands, build.heights);\n", "")], "a group is sized before it is appended, in the row loop"), # Predicting before there is anything to predict from. Dropping the guard does not write a # wrong number — it throws on `null.head`, and the throw happens while the very first batch is # being built, before a single row reaches the ledger. ("the guard against sizing without heights is dropped", - [(" function applyIntrinsicSize(section, rows, bands, heights) {\n if (!heights) return;\n", + [(" function applyIntrinsicSize(section, rows, bands, heights) {\n if (!heights) return 0;\n", " function applyIntrinsicSize(section, rows, bands, heights) {\n")], "declining to size without heights is not a crash"), diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-ledger-lifecycle.py b/plugins/avatasia/mmc-trajectory/tests/mutate-ledger-lifecycle.py new file mode 100644 index 00000000..d7c46397 --- /dev/null +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-ledger-lifecycle.py @@ -0,0 +1,493 @@ +"""Break the per-group lifecycle in the shipped client and require verify-ledger-lifecycle.mjs +to report each break by name. + +The lifecycle has two halves that pull in opposite directions, and a version of this that keeps +one and loses the other is worse than not doing it at all: + + - a group the reader is not on must not be holding its rows, or one session of about 50,600 + records is 463,122 DOM nodes and 785 to 820 MB of the renderer's private memory; + - a group that has given up its rows must still be the height it was, or the document under + the reader loses everything they have scrolled past — measured at 79,288 px for the groups + around the viewport and 257,378 px more for the viewport's own group. + +Each mutation below breaks one of those, or one of the transitions between them, and the guard +has to name the rule it broke. A mutation that is only "the file no longer parses" proves +nothing, so the guard has to exit non-zero *and* report the rule by name. + +Anchors are multi-line and index.html is LF, so they are written with bare \\n and the file is +read and written with newline='' — matching on normalised text while preserving the file's own +endings. +""" +import io +import os +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +CLIENT = os.path.join(os.path.dirname(HERE), "miniapp", "client", "index.html") +PROBE = os.path.join(HERE, "verify-ledger-lifecycle.mjs") + +# Every anchor below is text the rule is written on — a statement, a condition, an argument +# list. Anchoring on prose would mean an edit that merely rewords a comment reported as drift, +# and an edit that breaks the rule without touching the prose went unnoticed. +# +# `expect` is the name of the assertion the guard has to report. It is a list where one break +# owns more than one rule: removing the class removal breaks both the state the reader sees +# and the rule that says the state is turned off. +MUTATIONS = [ + # The build is not a visit. The poll runs every eight seconds whether the reader is there + # or not, so a build that stamps the reader-idle lease renews it forever: the fifteen + # second idle timeout can never pass, and the group the reader left keeps its row DOM for + # the rest of the session. + ("a rebuild renews the reader-idle lease of a group the reader has left", + [(" var holds = ledgerHoldsRows(key, build.at === 1);\n", + " var holds = ledgerHoldsRows(key, build.at === 1);\n" + " ledgerStateOf(key).lastSeen = Date.now();\n")], + ["rebuilding an offscreen group does not renew its reader-idle lease", + "the build does not write the reader-idle lease"]), + + # The window is anchored on one group, and that group holds its rows whatever the reader + # is doing — so it has to be counted among the groups that hold their rows. Counted with + # its own lease instead of the current time, a group the reader left long ago is the + # oldest one there is, and the cap reaches for it first. + ("the window's first group is released with the others", + [(" live.push({ key: kept[j], at: now });\n", + " live.push({ key: kept[j], at: ledgerStateOf(kept[j]).lastSeen });\n")], + "the window's first group is one of the groups that hold their rows"), + + # The same group left out of the count altogether: it goes on holding its rows past the + # bound, so the bound is about one set of groups while the rows on screen are another. + ("the window's first group is not counted among the groups that hold their rows", + [(" ledgerStateOf(kept[j]).live = true;\n" + " live.push({ key: kept[j], at: now });\n", + "")], + "the window's first group is one of the groups that hold their rows"), + + # The idle timeout is what releases a group the reader has walked away from. Without it a + # group stays live for as long as the session is open, and the bound below becomes the only + # thing standing between the reader and 785 MB. + ("a group the reader has left never gives its rows up", + [(" if (now - state.lastSeen >= LEDGER_IDLE_MS) {\n", + " if (false) {\n")], + "a group the reader has left gives its rows up once the idle time has passed"), + + # The bound is what makes the memory bounded rather than merely reduced: a reader who + # scrolls through three hundred groups in a minute has been on all of them. + ("the number of groups holding their rows is never bounded", + [(" if (live.length <= LEDGER_LIVE_GROUPS) return doomed;\n", + " if (true) return doomed;\n")], + "the number of groups holding their rows is bounded"), + + # The oldest goes first, which is the order the reader would choose. Sorting the other way + # releases the group the reader is on and keeps the one they left long ago. + ("the newest group is released instead of the oldest", + [(" live.sort(function (a, b) { return a.at - b.at; });\n", + " live.sort(function (a, b) { return b.at - a.at; });\n")], + "the oldest groups are the ones that go when the bound is passed"), + + # The height is the reader's place in the session. `contain-intrinsic-size` sizes a group + # the browser has skipped, and a group near the viewport is rendered — so without an + # explicit height the box collapses to its 39 px header and the document shrinks under + # the reader. + ("a released group loses the height it is given", + [(" section.style.minHeight = Math.round(height) + 'px';\n", + " section.style.minHeight = '';\n")], + "a released group keeps the height it is given"), + + # A group with no measured height gets no reserved height either. Writing 0px would be + # worse than writing nothing: a group sized 0px can never come back into the viewport, so + # it is never re-rendered, and the ledger stays collapsed for good. + ("a group with no height to keep is given one anyway", + [(" if (!(height > 0)) return;\n", + " if (false) return;\n")], + "a group with no height to keep is not given one"), + + # The rows come back when the reader reaches the group. This is the whole of what the + # reader is owed for having given them up. + ("the rows do not come back when the reader reaches the group", + [(" fillGroupRows(list, group);\n", "")], + "a group the reader reaches puts its rows back"), + + # The released state is what the reader sees while the group is empty. + ("the released state is left on after the rows come back", + [(" section.classList.remove('is-released');\n", "")], + ["a group that has its rows back drops the released state", + "the released state is turned off when the rows come back"]), + + # The reserved height is given back when the group has its rows again, or the group is + # taller than the rows it holds. + ("the reserved height is not given back after the rows come back", + [(" section.style.minHeight = '';\n", "")], + "a group that has its rows back gives its reserved height back"), + + # A group that is already holding its rows is filled by the build and then again by the + # observer, and the second pass appends every row a second time. + ("a group that is already holding its rows is filled a second time", + [(" var alreadyLive = section.getAttribute('data-rows') === 'live';\n", + " var alreadyLive = false;\n")], + "a group that is already holding its rows is not filled a second time"), + + # The first group of a window is where the reader arrives, and every other group's height + # is predicted from it. If it does not hold its rows there is nothing to predict from. + ("the first group of a window does not hold its rows", + [(" if (isFirst) return true;\n", + " if (isFirst) return false;\n")], + "the first group of a window always holds its rows"), + + # The anchor is read out of the group the reader is on. A released group has no rows, and + # an anchor read from the next group down that still has rows is an anchor to the wrong + # place — the reader is moved to wherever that group starts. + ("the anchor is read without filling the group the reader is on", + [(" if (sections[s].getAttribute('data-rows') !== 'live') fillLedgerGroup(sections[s]);\n", + "")], + "the group the reader is on is filled before the anchor is read out of it"), + + # The build asks for no rows for a group it is not holding. This is the one that carries + # the whole memory win: asking for them and then hiding them is the same cost. + ("the build asks for the rows of every group", + [(" var bands = fillGroupRows(holds ? list : null, group);\n", + " var bands = fillGroupRows(list, group);\n")], + "the build asks for no rows for a group it is not holding"), + + # A group the build is not holding is given a height to keep. + ("a group the build is not holding is given no height to keep", + [(" holdLedgerHeight(section, heldLedgerHeight(key, group.entries.length - bands, bands, predicted, build.heights));\n", "")], + "a group the build is not holding is given a height to keep"), + + # The two states are what the rest of the page reads: the build says which, and the + # observer fills the other. + ("a group the build is holding does not say so", + [(" section.setAttribute('data-rows', 'live');\n", "")], + "a group the build is holding says so"), + ("a group the build is not holding does not say so", + [(" section.setAttribute('data-rows', 'released');\n", "")], + "a group the build is not holding says so too"), + + # The released state is what the reader sees while the group is empty, so the rule that + # turns it on has to exist in the build and not only in the stylesheet. + ("the released state is never turned on", + [(" section.classList.add('is-released');\n", "")], + "the released state is turned on by the build"), + + # The bands of a released group are counted even though its rows are not built: its height + # is predicted from how many bands it has, and a prediction that ignored them would be + # short by every band in it. + ("the bands of a released group stop being counted", + [(" if (isBand) bands += 1;\n", + " if (isBand && list) bands += 1;\n")], + "the bands of a released group are still counted"), + + # A compaction record is a band, not a row, and the test that says so has to be the one the + # branch uses. A version that computes the test and then ignores it says the same word and + # renders the record as a row — which is why this break is in the branch rather than in the + # word. + ("a compaction record is rendered as a row instead of a band", + [(" if (isBand) {\n", " if (false) {\n")], + "a compaction record is rendered as a band, not as a row"), + + # The map that says which group each section stands for has to go with the sections. A + # section that is no longer in the table must not be filled when the observer reports it + # as in view — it would be filled with rows that belong to a table that no longer exists. + ("the ledger does not forget the groups it just emptied", + [(" ledgerContent.clear();\n\n if (state.loading) {\n" + " ledger.appendChild(buildSkeleton());\n", + " if (state.loading) {\n ledger.appendChild(buildSkeleton());\n")], + "the ledger drops what it knew about the groups it just emptied"), + + # A turn key means nothing across two sessions, so the groups of the session being left + # have to go with it. + ("the groups of the session being left are not forgotten", + [(" ledgerState.clear();\n ledgerContent.clear();\n", + " ledgerContent.clear();\n")], + "the groups of the session being left are forgotten with it"), + + # The height a released group keeps has to describe the content it holds now. A group is + # keyed by its turn, and the turn keeps its key while its content changes — a late reply, a + # band from a compaction — so the height taken when the group was last on screen describes + # content that is no longer there. Without the difference being charged, the group keeps + # the height of a content it no longer holds, and the document under the reader comes up + # short by exactly the rows that were added. + ("a released group does not bring the height it keeps up to date", + [(" var held = state.height + addedRows * heights.row + addedBands * heights.band;\n", + " var held = state.height;\n")], + "a released group brings the height it keeps up to date with the rows it gained"), + + # A height that does not say what it was a measurement of cannot be brought up to date: + # there is no difference to charge, and the group keeps a height that describes nothing. + ("the height a group was measured at stops recording what it was holding", + [(" var measuredBands = group.bands || 0;\n" + " state.heightRows = group.entries.length - measuredBands;\n" + " state.heightBands = measuredBands;\n", + "")], + "the height a group was measured at records what it was holding"), + + # A band is not a row. It has its own height, and a group that gained one is taller by a + # band and not by a row — so a band that is not counted is a band that is not charged. + ("the band a group was measured at stops being recorded", + [(" state.heightBands = measuredBands;\n", " state.heightBands = 0;\n")], + "a released group is charged for the band it gained"), + + # The measurement has to record rows and bands as two counts. A band is an entry as well as + # a row, so recording the entry count charges every band once as a row and once as a band: + # a band would be charged 36 px and 38 px, and the box would stand a whole row taller than + # the content it is holding — the document lying about its own height in the other + # direction. The guard's band cases add a band without removing a row, so this shows up. + ("the measurement records the entry count instead of the row count", + [(" state.heightRows = group.entries.length - measuredBands;\n", + " state.heightRows = group.entries.length;\n")], + ["a released group is charged for the band it gained", + "a released group that lost a row and gained a band is charged for both"]), + + # The measured height belongs to the observer: it is taken when the group is really laid + # out. A build that stamped one would be writing down a height it never measured, and the + # next rebuild would read its own guess back as if it were a measurement. + ("the build writes the height it computed", + [(" holdLedgerHeight(section, heldLedgerHeight(key, group.entries.length - bands, bands, predicted, build.heights));\n", + " ledgerStateOf(key).height = predicted;\n" + " holdLedgerHeight(section, heldLedgerHeight(key, group.entries.length - bands, bands, predicted, build.heights));\n")], + ["a rebuild does not change the height a group was measured at", + "the build does not write a measured height"]), + + # A group that has been on screen keeps the height it was measured at. The prediction is + # priced from medians, and a group's real height runs to several times the median one, so + # falling back on the prediction trades a real height for an average one. + ("a released group is given the prediction instead of the height it was measured at", + [(" holdLedgerHeight(section, heldLedgerHeight(key, group.entries.length - bands, bands, predicted, build.heights));\n", + " holdLedgerHeight(section, predicted);\n")], + "a released group whose content has not changed keeps the height it was measured at"), + + # The prediction prices the bands on their own height. A band is an entry as well as a row, + # so a prediction that takes the entry count as the row count prices every band twice: once + # at the row's height and once at the band's own. The two are close — 36 px and 38 px on a + # real session — so the box stands a whole row taller than the content it is holding, which + # is the document lying about its own height in the other direction. + # + # The two call sites are separate breaks: the one the pass after the calibration makes, and + # the one the build makes for every group it writes. + ("the prediction charges a band as a row as well", + [("var dormantPredicted = applyIntrinsicSize(dormant[d], dormantRows, content.bands, build.heights);", + "var dormantPredicted = applyIntrinsicSize(dormant[d], content.entries.length, content.bands, build.heights);")], + "a band is priced once in the predicted height"), + + ("the build charges a band as a row as well", + [("var predicted = applyIntrinsicSize(section, group.entries.length - bands, bands, build.heights);", + "var predicted = applyIntrinsicSize(section, group.entries.length, bands, build.heights);")], + "the prediction of a group with a band does not charge the band as a row"), + + # The pass that follows the calibration sizes the groups the build released, and what it + # writes is the height that is HELD — the measured one, brought up to date with what the + # group holds now. Writing the prediction there undoes the rule on every group of the first + # batch, and in the default view the first batch is the whole window: the page opens on the + # newest 200 records, and 200 records span fewer turns than LEDGER_BATCH_GROUPS. + ("the calibration overwrites the height a released group keeps", + [("holdLedgerHeight(dormant[d], heldLedgerHeight(content.key, dormantRows, content.bands, dormantPredicted, build.heights));", + "holdLedgerHeight(dormant[d], dormantPredicted);")], + "a released group keeps the height it was measured at in the first batch"), + + # The build makes the first group of a window live, and it is laid out for real, but the + # build never measures — the measurement is the observer's. So the group the build made + # live has no height of its own until the observer looks at it, and the visit used to + # return early because the rows were already there. The group then fell back on the + # prediction when it was released: three rows and a band laid out at 1,000 px reserved + # 208 px, with not one row or band changed. + ("a group the build made live never records the height it was measured at", + [(" if (!alreadyLive || !state.height || ledgerLayoutChanged(section, state)) measureLedgerGroup(section, group);\n", + " if (!alreadyLive || ledgerLayoutChanged(section, state)) measureLedgerGroup(section, group);\n")], + "a group built live keeps its measured height when a wider window releases it"), + + # A height is a measurement of a layout, and a layout can change without the content + # changing: the window is resized, the text scale is turned up, the group's own chrome + # becomes taller. The rows and the bands stay exactly the same and the height is a + # different one — and the height a released group keeps is the remembered one unchanged, + # because the counts it was measured at have not changed either. So a group laid out at + # 1,500 px kept the 1,000 px it was measured at, and the document under the reader came + # up 500 px short of the group it stood for. + # + # The break takes away the refresh and nothing else: the group is still measured when it + # has no height of its own, and the rows are still built once. + ("a live group never refreshes its measurement after its layout changes", + [(" if (!alreadyLive || !state.height || ledgerLayoutChanged(section, state)) measureLedgerGroup(section, group);\n", + " if (!alreadyLive || !state.height) measureLedgerGroup(section, group);\n")], + "a live group refreshes its measurement after its layout changes"), + + # The window is resized and the client is told: the groups that are on screen have to be + # measured again, because a resize changes their layout without changing a row or a band + # in them. A client that does not re-measure them keeps the height it measured them at — + # a height of the layout before the resize — and when the cap releases the group, the + # height it reserves is that stale number, so the document under the reader comes up short + # by the difference. + # + # The break takes away the measurement the resize drives and nothing else: the rows are + # still built once, the cap still releases the same groups, and the measurement the visit + # path takes is untouched. + # + # The boundary, which this break does not touch either: a group that was off screen when + # the layout changed, and is never visited again, keeps the height it was last really + # measured at. Off screen the browser answers with the group's own contain-intrinsic-size, + # and that number is a prediction — a prediction written down as a measurement is a guess + # the next rebuild reads back as a fact. + ("a live group never refreshes its measurement when the window is resized", + [(" measureLedgerGroup(section, group);\n", "")], + "a live group refreshes its measurement when the window is resized"), + + # The bound is about what the reader is holding at this moment, not only about what a + # rebuild leaves behind. Without the cap on the visit path, a reader who walks through + # more groups than the cap allows between two rebuilds is holding all of them. + ("reader visits do not enforce the cap on the groups holding their rows", + [(" releaseLedgerRows(releaseStaleGroups([ledgerFirstGroupKey(), group.key]));\n", "")], + "reader visits cannot increase the held row groups past the ceiling between rebuilds"), + + # The visit path holds on to the group the reader has just reached. It has to hold on to + # the window's first group as well: that group holds its rows whatever the reader is + # doing, it is where the reader arrives, and every other group's height is predicted from + # it. A visit that protected only the group the reader had just reached let the window's + # first group go — the idle timeout took it, or the cap reached for it as the oldest one — + # and the group the heights are predicted from lost its rows while the reader was on the + # page. + ("the visit path does not protect the window's first group", + [(" releaseLedgerRows(releaseStaleGroups([ledgerFirstGroupKey(), group.key]));\n", + " releaseLedgerRows(releaseStaleGroups([group.key]));\n")], + ["the window's first group keeps its rows after a round of visits", + "the window's first group keeps its rows when the reader comes back to another group"]), + + # A group the cap released gives up its rows at once. The rows are the DOM, and a group + # that has given them up must not still be holding them. + ("a group the cap released keeps its rows", + [(" if (list) clear(list);\n", "")], + "a group the cap released holds no rows"), + + # And it keeps the height it has now. The height is the reader's place in a session they + # have scrolled through, and a group that has given up its rows without keeping it takes + # the document down under them. + ("a group the cap released keeps no reserved height", + [(" holdLedgerHeight(section, heldLedgerHeight(content.key, rows, bands, measured, ledgerCalibration));\n", "")], + "a group the cap released keeps the height it has now"), + + # The height a released group keeps has to describe the content it holds now, and the + # visit path has no calibration of its own to price the difference with. Passing no + # calibration writes the old measurement back unchanged: a group measured at 1,000 px with + # three rows in it and holding four rows by the time it is released on the visit path keeps + # 1,000 px instead of 1,036 px, and the document under the reader comes up short by the row + # that was added. + # + # The group that witnesses it is one the reader has left: it is off screen, so the browser + # skips it and what it reads is the prediction, and the measured height brought up to date + # with the row that was added is the only answer that is not a guess. A group the reader + # is on is measured again when its layout changes, and its measurement is the current one. + ("the visit path writes back the height the group was measured at", + [(" holdLedgerHeight(section, heldLedgerHeight(content.key, rows, bands, measured, ledgerCalibration));\n", + " holdLedgerHeight(section, heldLedgerHeight(content.key, rows, bands, measured, null));\n")], + "a group released off screen keeps the height it was measured at, brought up to date"), + + # The rebuild is one of the two places that decides the bound, and the fixture's rebuild() + # calls the shipped function that does it. A rebuild that stopped releasing the groups the + # reader had left came out with more groups holding their rows than the cap allows, and + # every assertion in the guard still passed. + ("the rebuild does not release the groups that have to give up their rows", + [(" releaseStaleGroups([turnKey(groups[0].turn)]);\n", "")], + "after a rebuild, the number of groups holding their rows is at most the cap"), + + # And the call site itself: the rebuild asks for the release before it writes the table. + ("the rebuild does not ask for the release before it writes the table", + [(" releaseStaleGroupsBeforeRebuild(groups);\n", "")], + "the rebuild releases the groups that have to give up their rows before it writes the table"), + + # The groups the caller holds on to are counted after the ones the reader has visited, so + # that a tie the clock cannot break cannot put them at the head of the queue. The cap + # reaches for the oldest, and the group the reader has just reached must not be the one it + # reaches for. Counted first, the window's first group is the head of the queue and the + # cap takes it. + ("the groups the caller holds on to are counted before the ones the reader has visited", + [(" for (var j = 0; j < kept.length; j += 1) {\n" + " ledgerStateOf(kept[j]).live = true;\n" + " live.push({ key: kept[j], at: now });\n" + " }\n", ""), + (" ledgerState.forEach(function (state, key) {\n", + " for (var j = 0; j < kept.length; j += 1) {\n" + " ledgerStateOf(kept[j]).live = true;\n" + " live.push({ key: kept[j], at: now });\n" + " }\n" + " ledgerState.forEach(function (state, key) {\n")], + "the window's first group keeps its rows when the clock cannot break the tie"), + + # The observer is registered by the build, and the build is what re-registers it after + # every rebuild: a section from the previous table is not in `ledgerContent` any more, so + # an observer left pointing at the old table never sees the new one. A build that stops + # watching leaves the reader's next visit unobserved, and the height the group keeps + # after that is the height of a layout that is gone — the group stands 500 px shorter + # than the content it is holding. + # + # The refresh that a resize drives cannot cover for it: the build sized the groups of its + # first batch by the height they are really at, so the number the group reads back is the + # number it was given, and the refresh skips it as a group that is sized rather than laid + # out. The observer's visit is the only thing that measures it again. + ("the build does not observe the groups it rebuilt", + [(" watchLedgerGroups();\n", "")], + ["resize and a rebuilt observer preserve the current layout through release", + "a rebuilt observer watches the group the reader is on, not the one it replaced"]), + + # The other half of the same rule, and the one a deletion cannot reach: an observer that + # is registered once and never pointed at the new table is still watching the sections of + # the previous table. The build's watcher disconnects the observer before it observes the + # new sections, and that disconnect is the whole of what makes the second build's + # observer the current one. Without it the observer's visit never reaches the group the + # reader is on — the section that replaced it is not one the observer knows — so the group + # keeps the height of the layout before the resize, and the document under the reader comes + # up short by exactly the difference. + ("the build observes the groups it rebuilt only once", + [(" } else {\n ledgerObserver.disconnect();\n }\n", + " } else {\n return;\n }\n")], + ["resize and a rebuilt observer preserve the current layout through release", + "a rebuilt observer watches the group the reader is on, not the one it replaced"]), +] + +original = io.open(CLIENT, encoding="utf-8", newline="").read() + + +def run(): + proc = subprocess.run(["node", PROBE], cwd=HERE, + capture_output=True, text=True, encoding="utf-8", errors="replace") + return proc.returncode, proc.stdout + proc.stderr + + +ok = True +code, baseline = run() +base_ok = code == 0 and "0 failed" in baseline +print(("ok " if base_ok else "FAIL ") + "baseline green") +ok = ok and base_ok + +caught = [] +for title, edits, expect in MUTATIONS: + for old, _ in edits: + hits = original.count(old) + if hits != 1: + raise SystemExit("FAIL anchor for %r matched %d times" % (title, hits)) + names = expect if isinstance(expect, list) else [expect] + try: + mutated = original + for old, new in edits: + mutated = mutated.replace(old, new, 1) + io.open(CLIENT, "w", encoding="utf-8", newline="").write(mutated) + during_code, during = run() + named = all(n in during for n in names) + red = during_code != 0 + # A guard that crashes is not a guard that caught anything: the rule has to be reported + # by name, or the mutation only proved that the file no longer parses. + print(("ok " if (named and red) else "FAIL ") + "guard catches: " + title) + if not named: + print(" no assertion reported %r" % (names,)) + for line in during.splitlines()[-4:]: + print(" " + line.strip()) + elif not red: + print(" the rule was reported but the guard still exited 0") + caught.append(named and red) + finally: + io.open(CLIENT, "w", encoding="utf-8", newline="").write(original) + +restored = io.open(CLIENT, encoding="utf-8", newline="").read() +after_code, after = run() +after_ok = after_code == 0 and "0 failed" in after +print(("ok " if restored == original else "FAIL ") + "client restored byte for byte") +print(("ok " if after_ok else "FAIL ") + "green again after restore") + +sys.exit(0 if (ok and all(caught) and restored == original and after_ok) else 1) diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-list-cache-ttl.py b/plugins/avatasia/mmc-trajectory/tests/mutate-list-cache-ttl.py index 26b63321..7e4d0999 100644 --- a/plugins/avatasia/mmc-trajectory/tests/mutate-list-cache-ttl.py +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-list-cache-ttl.py @@ -153,28 +153,41 @@ # above still passes and the page is simply broken on every database written before child # sessions existed. The assertion that catches it is the one that builds such a database here. ("a database without the parent column loses every title and kind, not just the parent", - [(""" let stmt; - try { - stmt = db.prepare('select session_id, title, session_kind, parent_session_id from local_runtime_sessions where session_id = ?'); - } catch { + [(""" let stmt = null; + for (const sql of SESSION_TITLE_SELECTS) { try { - stmt = db.prepare('select session_id, title, session_kind from local_runtime_sessions where session_id = ?'); + stmt = db.prepare(sql); + break; } catch { - return result; + // This database does not have that column yet; ask for less. } } -""", " const stmt = db.prepare('select session_id, title, session_kind, parent_session_id from local_runtime_sessions where session_id = ?');")], + if (!stmt) return result; +""", " const stmt = db.prepare(SESSION_TITLE_SELECTS[0]);")], "a database WITHOUT the column keeps the title and the kind, and only loses the parent"), + # ---- the ladder: one rung at a time ---- + # + # The read asks for the columns a current runtime writes and falls back, one column at a time, + # to the ones an older database has. A rung that is missing is not a rung that asks for less: + # the database above it lands on the empty catch instead, and the page shows no titles and no + # kinds at all. Both of these remove one rung, and each is caught by the assertion about the + # database shape that rung was there for. + ("the rung that reads the parent without the purpose is dropped", + [(" 'select session_id, title, session_kind, parent_session_id from local_runtime_sessions where session_id = ?',\n", "")], + "a database WITHOUT the purpose column still reports the title, the kind and the parent"), + + ("the rung that reads the purpose without the workspace is dropped", + [(" 'select session_id, title, session_kind, parent_session_id, purpose from local_runtime_sessions where session_id = ?',\n", "")], + "a database WITHOUT the workspace column still reports the title, the kind, the parent and the purpose"), + ("the parent row is read but never mapped", - [(" parent: typeof row.parent_session_id === 'string' && row.parent_session_id !== ''\n" - " ? row.parent_session_id\n" - " : null,\n", "")], + [(" parent: parentNames ? parent : null,\n", "")], 'the parent row is mapped, and an empty id is no parent rather than a parent named ""'), - ("the descriptor is built without the parent", - [(" descriptor.sessionParentId = meta.parent;", "")], - "the descriptor carries the parent onto the row the page receives"), + ("the row is built without the parent", + [(" sessionParentId: meta ? meta.parent : null,", "")], + "the row the page receives carries the parent, on every row"), # This one is the most likely to be reintroduced by accident: it is a whole-field removal in # a function whose job is to decide what may leave the service, and nothing else in it fails. @@ -182,12 +195,32 @@ [(" sessionParentId: value.sessionParentId ?? null,", "")], "the field survives the strip, with null as the floor"), - # The direction that looks right and is not: marking every row. Every string the guard looks - # for is still present, and the list reads as though every conversation were a sub-agent run. - ("the mark is applied to every session rather than the branches", - [(" var branch = session && session.sessionParentId ? BRANCH_MARK : '';", - " var branch = BRANCH_MARK;")], - "the page decides the marker by the field and not unconditionally", + # The parent column is a foreign key into the runtime database's own session table, and the + # page groups its runs by that name -- so a value that names no session there is a name the + # list must not publish. No regex over the source can catch this one: the query ran, it + # found nothing, and the only thing that shows it is a database built to have that hole. + ("a parent that names no session is published as a parent anyway", + [(" const parentNames = parent ? Boolean(stmt.get(parent)) : false;", + " const parentNames = Boolean(parent);")], + "a parent that names no session is not published as a parent"), + + # The direction that looks right and is not: the parent field is read, but not as the key + # the rows are grouped under. Every conversation then hangs the same sub-agents, and the run + # pick of every conversation lists the same thirteen of them -- so a reader picks a run that + # belongs to somebody else's conversation, and the page shows it as their own. + ("the parent field is read, but not as the key the rows are grouped under", + [(" var bucket = grouped[child.sessionParentId];", + " var bucket = grouped['mvs_one_parent_for_everything'];")], + "the page decides which rows are sub-agents by the parent field and not unconditionally", + CLIENT), + + # The same pair, for the kind badge: applied to every row, and applied to none. The first one + # still contains 轻量 on every line, so every check that asks "does the row say 轻量" passes — + # and every ordinary conversation now claims to be a lightweight exchange. + ("the kind badge is applied to every session", + [(" var tag = session ? sessionTagBadge(session.sessionTag) : '';", + " var tag = sessionTagBadge('lightweight');")], + "the page decides the kind badge by the field and not unconditionally", CLIENT), ] diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-mutation-contracts.py b/plugins/avatasia/mmc-trajectory/tests/mutate-mutation-contracts.py new file mode 100644 index 00000000..eab7caa6 --- /dev/null +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-mutation-contracts.py @@ -0,0 +1,387 @@ +"""The control that proves the name contract is enforced, and that it cannot be silenced. + +`verify-mutation-contracts.py` is a guard about the OTHER mutation controls: it reads each of +them and says whether a case is judged by the NAME of the rule that case broke, or by one +reported name standing in for it. A guard that cannot fail is not a guard, so this file breaks +that guard five ways and requires it to fail by name every time: + + 1. the decision compares the case's own expected name with the FIRST name the guard reported; + 2. the decision drops the case's expected name and takes the first name the guard reported; + 3. the case table drops the expected-name field altogether; + 4. the set of controls the guard derives is empty; + 5. the set of controls the guard derives has been shrunk to one. + +Cases 1 to 3 are the relaxations of a real control -- `mutate-search-endpoint.py`, one of the two +the reviewer found -- and they are undone byte for byte. Cases 4 and 5 are the silencing: they +run a COPY of the guard from a directory that holds fewer controls than the package does, which +is what a guard that reads a hand-written list, or one whose list has drifted, looks like from +the inside. A guard that reported an empty set as clean would pass both of them, and the two +cases are here so that it cannot. + +Every case is caught BY NAME. The guard's failure lines are collected and the case's expected +rule name has to be among them: a guard that went red for some other reason has not shown that +it owns this mutation, which is the same rule this file is about. +""" + +import ast +import hashlib +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +HERE = Path(__file__).resolve().parent +GUARD = HERE / "verify-mutation-contracts.py" +TARGET = HERE / "mutate-search-endpoint.py" + +# The rules the guard reports, spelled the way it spells them. +RULE_MEMBERSHIP = "a mutation control decides a case by membership, not by one reported name" +RULE_ANCHOR = "the two reviewed mutation controls decide their cases by membership" +RULE_DERIVED = "the set of mutation controls is derived from the package and is not empty" +RULE_CLAIM = "a control that claims it caught cases by name judges them by membership" + +# The two controls the guard anchors on. They are copied into the shorter trees below so that the +# anchor holds there and the rule under test is the only one that can fail. +ANCHOR = ("mutate-search-endpoint.py", "mutate-search-segments.py") + +# The claim, as a control that does not honour it would print it. It goes at the END of a control's +# source, so it is a print of its own and nothing else in the file changes. +CLAIM_PRINT = 'print("all mutations caught, each by name")' + +# The decision that is correct, and the two that are not. The case table is not touched by the +# first two: every case keeps its expected rule name, and only the decision changes. +DECISION_FROM = r""" # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + if expect not in names: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") +""" + +DECISION_SINGLE = r""" # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + first = names[0] + if first != expect: + print(f"MISSED {label}: the probe named {names} instead of {expect}") + missed.append(label) + continue + print(f"caught {label}\n -> {first}") +""" + +DECISION_FIRST = r""" # The case names the rule it is about, so the probe has to name THAT rule and not some + # other one that happened to go red in the same run: a probe that went red elsewhere has + # not shown that this guard owns this mutation. + first = names[0] + print(f"caught {label}\n -> {first}") +""" + + +def read_exact(path): + """Normalise to LF for matching, and report the file's own newline so it can be put back.""" + data = path.read_bytes() + text = data.decode("utf-8").replace("\r\n", "\n") + return text, ("\r\n" if b"\r\n" in data else "\n") + + +def write_exact(path, text, newline): + if newline == "\r\n": + text = text.replace("\n", "\r\n") + path.write_bytes(text.encode("utf-8")) + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def reported_rules(output): + """The rules the guard named. + + Two formats, as in `mutate-search-endpoint.py`: ` - ` and `FAIL `. A case is + only caught when the rule it names is among them. + """ + names = [] + for line in output.splitlines(): + text = line.strip() + if text.startswith("- "): + names.append(text[2:].strip()) + elif text.startswith("FAIL "): + names.append(text[5:].strip()) + return names + + +def run_guard(script=None, cwd=None): + """Run the guard, or a copy of it, and report what it said.""" + command = [sys.executable, str(script if script is not None else GUARD)] + proc = subprocess.run( + command, + cwd=str(cwd if cwd is not None else HERE), + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + ) + return proc.returncode, (proc.stdout or "") + (proc.stderr or "") + + +# -------------------------------------------------------------------------------------- +# the three relaxations of a real control +# -------------------------------------------------------------------------------------- + +def relax_decision(text, replacement): + """Replace the decision that judges a case, and nothing else.""" + if DECISION_FROM not in text: + raise SystemExit("the decision this control relaxes is not in the file any more") + return text.replace(DECISION_FROM, replacement, 1) + + +def drop_expected_field(text): + """Delete the expected-name field from every case, and the decision that read it. + + The field is the fourth element of every tuple in the case table, and it is found through + the tree rather than through a pattern: the line each one sits on is read off the parse, so + a case whose text happens to look like another case's text cannot be mistaken for it. + """ + tree = ast.parse(text) + table = None + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + for target in node.targets: + if isinstance(target, ast.Name) and target.id == "MUTATIONS": + table = node.value + if table is None: + raise SystemExit("the case table this control edits is not in the file any more") + + drop = set() + for element in table.elts: + if len(element.elts) < 4: + raise SystemExit("a case no longer carries an expected rule name") + field = element.elts[3] + if field.end_lineno != field.lineno: + raise SystemExit("an expected rule name now spans more than one line") + drop.add(field.lineno) + + kept = [line for number, line in enumerate(text.split("\n"), 1) if number not in drop] + out = "\n".join(kept) + out = out.replace("for label, old, new, expect in MUTATIONS:", + "for label, old, new in MUTATIONS:") + out = relax_decision(out, DECISION_FIRST) + # The name has to be gone as a NAME, not as a word: the file's own prose says "expect". + for node in ast.walk(ast.parse(out)): + if isinstance(node, ast.Name) and node.id == "expect": + raise SystemExit("the expected rule name is still read after the field was dropped") + return out + + +RELAXATIONS = [ + ("the decision compares the case's expected name with the first name the guard reported", + RULE_MEMBERSHIP, + lambda text: relax_decision(text, DECISION_SINGLE)), + ("the decision drops the case's expected name and takes the first name the guard reported", + RULE_ANCHOR, + lambda text: relax_decision(text, DECISION_FIRST)), + ("the case table drops the expected-name field", + RULE_ANCHOR, + drop_expected_field), +] + + +# -------------------------------------------------------------------------------------- +# the two silences +# -------------------------------------------------------------------------------------- + +def run_in_a_shorter_tree(label, expect, keep): + """Run a COPY of the guard from a directory that holds fewer controls than the package. + + The copy is the guard's own source, unchanged. What differs is the directory it sits in, and + that is the point: the set of controls it derives comes from where it stands, so a guard + that read a list, or one whose list had drifted, would find fewer controls here and report + on fewer. This case requires it to fail instead. + """ + workspace = tempfile.mkdtemp(prefix="mutation-contracts-") + try: + shutil.copy(GUARD, Path(workspace) / GUARD.name) + for name in keep: + shutil.copy(HERE / name, Path(workspace) / name) + code, output = run_guard(Path(workspace) / GUARD.name, cwd=workspace) + names = reported_rules(output) + if code == 0: + print(f"MISSED {label}: the guard passed with {len(keep)} control(s) in reach") + return False + if expect not in names: + print(f"MISSED {label}: the guard named {names} instead of {expect}") + return False + print(f"caught {label}\n -> {expect}") + return True + finally: + shutil.rmtree(workspace, ignore_errors=True) + + +def non_claiming_control(output): + """The first control the guard reports as not judging by membership AND not claiming to. + + Both lists come out of the guard's own report, so nothing here is written out by hand: a + control added tomorrow is picked up without this file being edited. A control that claims to + catch its cases by name and does not judge them by membership is already reported by the + rule under test, so it cannot serve as the subject of this case. + """ + not_members = [] + claimers = [] + section = None + for line in output.splitlines(): + text = line.strip() + if text.startswith("by membership ("): + section = "members" + continue + if text.startswith("not by membership ("): + section = "not" + continue + if text.startswith("claiming to catch cases by name ("): + section = "claim" + continue + if not text: + continue + if not text.startswith("mutate-"): + section = None + continue + name = text.split()[0] + if section == "not": + not_members.append(name) + elif section == "claim": + claimers.append(name) + for name in not_members: + if name not in claimers: + return name + return None + + +def run_with_a_claim_in_a_shorter_tree(): + """Run a COPY of the guard against a control that claims to catch cases by name. + + The control is one the guard itself reports as not judging by membership, and the claim is + put back into it -- in the copy, never in the package. The guard has to fail, and it has to + name the rule: a guard that went red for another reason has not shown that this one bites. + """ + code, output = run_guard() + if code != 0: + print("the guard does not pass on the package as it stands; fix that first") + return False + target = non_claiming_control(output) + if target is None: + print("no control that does not judge by membership and does not claim to: " + "nothing for this case to prove") + return False + + workspace = tempfile.mkdtemp(prefix="mutation-contracts-claim-") + try: + shutil.copy(GUARD, Path(workspace) / GUARD.name) + for name in list(ANCHOR) + [target]: + shutil.copy(HERE / name, Path(workspace) / name) + path = Path(workspace) / target + text = path.read_text(encoding="utf-8") + path.write_bytes((text + "\n\n" + CLAIM_PRINT + "\n").encode("utf-8")) + code, output = run_guard(Path(workspace) / GUARD.name, cwd=workspace) + names = reported_rules(output) + if code == 0: + print(f"MISSED the claim that the cases were caught by name is put back into a " + f"control that does not judge by membership: the guard passed with " + f"{target} claiming to catch cases by name") + return False + if RULE_CLAIM not in names: + print(f"MISSED the claim that the cases were caught by name is put back into a " + f"control that does not judge by membership: the guard named {names} " + f"instead of {RULE_CLAIM}") + return False + print(f"caught the claim that the cases were caught by name is put back into a " + f"control that does not judge by membership\n -> {RULE_CLAIM}") + return True + finally: + shutil.rmtree(workspace, ignore_errors=True) + + +def main(): + original_bytes = TARGET.read_bytes() + original_digest = digest(TARGET) + missed = [] + unnamed = [] + + # The guard as it stands has to pass, or the cases below prove nothing. + code, output = run_guard() + if code != 0: + print("the guard does not pass on the package as it stands; fix that first") + print(output) + return 1 + print("baseline clean") + + text, newline = read_exact(TARGET) + for label, expect, relax in RELAXATIONS: + relaxed = relax(text) + if relaxed == text: + print(f"SKIP {label}: the relaxation changes nothing") + missed.append(label) + continue + write_exact(TARGET, relaxed, newline) + try: + code, output = run_guard() + finally: + TARGET.write_bytes(original_bytes) + if code == 0: + print(f"MISSED {label}: the guard still passed") + missed.append(label) + continue + names = reported_rules(output) + if not names: + print(f"UNNAMED {label}: the guard failed without naming a rule") + unnamed.append(label) + continue + if expect not in names: + print(f"MISSED {label}: the guard named {names} instead of {expect}") + missed.append(label) + continue + if TARGET.name not in output: + print(f"MISSED {label}: the guard did not name {TARGET.name}") + missed.append(label) + continue + print(f"caught {label}\n -> {expect}") + + # The guard cannot be silenced by the set it derives shrinking to nothing. + if not run_in_a_shorter_tree( + "the set of controls the guard derives is empty", + RULE_DERIVED, []): + missed.append("the set of controls the guard derives is empty") + + # Nor by the set shrinking to one control that happens to be correct. + if not run_in_a_shorter_tree( + "the set of controls the guard derives has been shrunk to one", + RULE_ANCHOR, ["mutate-search-endpoint.py"]): + missed.append("the set of controls the guard derives has been shrunk to one") + + # Nor by a control claiming to catch its cases by name while judging them by something else. + if not run_with_a_claim_in_a_shorter_tree(): + missed.append("the claim that the cases were caught by name is put back into a " + "control that does not judge by membership") + + if TARGET.read_bytes() != original_bytes: + print(f"NOT RESTORED {TARGET}") + return 1 + if digest(TARGET) != original_digest: + print(f"NOT RESTORED {TARGET}: the bytes are the same file but not the same hash") + return 1 + print("file restored byte-for-byte") + + if missed or unnamed: + if missed: + print(f"\n{len(missed)} mutation(s) got through") + if unnamed: + print(f"{len(unnamed)} mutation(s) failed without naming a rule") + return 1 + print(f"\nall {len(RELAXATIONS) + 3} relaxations caught, each by name") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/avatasia/mmc-trajectory/tests/mutate-parsed-rows-owner-escapes.py b/plugins/avatasia/mmc-trajectory/tests/mutate-parsed-rows-owner-escapes.py new file mode 100644 index 00000000..8818d306 --- /dev/null +++ b/plugins/avatasia/mmc-trajectory/tests/mutate-parsed-rows-owner-escapes.py @@ -0,0 +1,333 @@ +"""Mutation control for the owner escapes that outlived the removal of the prose contract. + +WHAT THIS FILE USED TO OWN, AND WHY IT IS SMALLER NOW. Most of the escapes this file held were +escapes of the READER: the machinery that read a number out of a sentence and compared it with the +number the service declares. That reader is gone. The service's comments no longer state the number +the package used to promise a reader -- the shipped comment names the RULE and the constants it +holds, and a guard that compared a comment with a constant was asserting that a piece of prose had +not gone stale. A control that kept those cases would have been holding a rule the package no +longer has, and the cases would have failed for a reason that has nothing to do with the guard. + +WHAT IS LEFT IS THE MACHINERY THE GUARD STILL USES TO OBSERVE THE SERVICE, and it is four shapes: + + * the PAGE. The guard reads the interval the page's own script hands its timer, so a page that + hands the timer a comma expression -- or a page whose inline script does not run at all -- is a + break the guard cannot see through. The rule that owns it says so by name. + * the SERVICE'S OWN BINDINGS. The guard reads the service's numbers from the module it imports, + so a declaration that is really a string, or a template literal impersonating one, is a break in + what the guard is about to read. + * THE COPY THE GUARD READS. The guard makes a copy of the service and imports it. A copy that + will not import leaves the guard with nothing to compare against, so it has to say that it could + not observe the numbers -- and it must not accuse any other rule. + * THE GUARD'S OWN SETUP. The cache starts its sweeper when the first row is written, so a service + that cannot start one throws in the middle of the guard's setup. A throw is a stack trace and not + the name of a rule, so the setup is wrapped, the failure is reported BY NAME, and the guard + stops there: without a cache there are no rows to read. + +THE FOURTH ONE IS NEW IN THIS ROUND (R29-F53). The guard used to let the throw escape, so a +service that could not start its sweeper produced a stack trace instead of a rule name, and a +review that reads a stack trace cannot tell which rule was broken. The case below breaks the +sweeper's creation, and the rule that owns it is the one that has to go red -- and it is the only +one that may. + +THE RULES ARE CHECKED THE WAY THE ROUNDS THAT FOUND THEM ASKED FOR: the owner has to be a +BEHAVIOUR, not a shape. The page's script is run in a sandbox and the delay comes from the timer +the page itself starts; the service's numbers come from the module's own bindings. + +Each case breaks one file and one rule, and a case counts as caught only when the guard reported +THAT rule -- by name -- and nothing else. The guard has to go red by NAMING a rule: a guard that +threw, or that went red without naming anything, has not shown that it owns this break. + +WHY THIS IS A SEPARATE FILE. `tests/mutate-parsed-rows.py` owns the thirty-six cases of the +cache's own behaviour, and its EXACT table is the single source of truth for which rules each of +those breaks may report. Those cases and that table are frozen: a round that changes product +behaviour would have to rewrite a table that is not its own to rewrite. A control whose table has +two owners is a table with no owner. So the escapes get their own control, and the suite runs one +more file. + +A CASE IS A TUPLE, AND THAT IS NOT A MATTER OF STYLE. `tests/verify-mutation-contracts.py` reads +this file's source to decide whether it judges its cases by membership -- whether the case's own +expected rule name is tested against the collection of names the guard reported -- and it reads a +case table as a LIST OF TUPLES. A case written as a dictionary is invisible to it, and a control +the guard cannot read is a control the guard cannot hold to the contract: this file was written +with dictionaries once, and the guard reported it as judging by the exit code alone. So the cases +are tuples, and a case that needs more than one edit carries its edits in parallel lists. + +TWO CASES BREAK THE GUARD ITSELF, and they are two shapes a control of this kind has to be able +to state: + + * a case can break the copy the guard reads the service's numbers from, so the guard cannot + observe them. The guard then has to FAIL LOUDLY -- by naming the rule it could not observe -- + and it must not accuse any other rule: a number the guard could not read is not a broken + rule. The exact set is what says so: a guard that also reported "the shipped default is + the interval the service declares" would have accused a rule the break did not break. + * a case can break TWO things at once, when the escape only exists in that combination. The + mis-accusation needs the observation to have failed AND the owner to compare anyway, so the + case breaks the copy and removes the gate, and the guard has to name the rule that says it + accuses nothing it could not observe. +""" +import json +import re +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +ROOT = HERE.parent +SERVER = ROOT / "miniapp" / "node" / "server.mjs" +CLIENT = ROOT / "miniapp" / "client" / "index.html" +PROBE = HERE / "verify-parsed-rows.mjs" + +# Each case: what it breaks, the files it breaks, the text before and after each of them, the +# rule that owns it, the EXACT set of rules that break may report, and anything the output has to +# say. +# +# The exact set is the isolation, and it is what the frozen control's table does for the other +# forty: a guard that also reported a rule this break did not break has accused another owner, +# and a reader cannot tell which rule had actually been broken. +# +# The anchor of every edit has to occur EXACTLY once in the file it edits. A control that replaces +# the first occurrence of a string that occurs twice edits a place it did not mean to edit, and +# the case then passes or fails for a reason that has nothing to do with the rule it is about. +MUTATIONS = [ + ( + "the page hands its timer a comma expression", + [CLIENT], + [" }, TRAJECTORY_POLL_INTERVAL);"], + [" }, (TRAJECTORY_POLL_INTERVAL, 8000000));"], + "the poll waits as long as the client declares", + ( + "the poll waits as long as the client declares", + "the time limit is longer than one poll, so an open tab never loses its rows", + ), + (), + ), + ( + "a template literal impersonates the proxy's declaration while the binding is a string", + [SERVER], + ["\nconst PARSED_ROWS_HEAP_PER_SOURCE_BYTE = 2;\n"], + ["\nconst annotation = `\nconst PARSED_ROWS_HEAP_PER_SOURCE_BYTE = 2;\n`;\n" + "const PARSED_ROWS_HEAP_PER_SOURCE_BYTE = '2';\n"], + "the heap proxy is a number the service declares", + ("the heap proxy is a number the service declares",), + (), + ), + ( + "the copy the guard reads the service's numbers from will not import", + [PROBE], + ["await writeFile(join(dir, 'server.mjs'), `${shipped}\\nexport { ${names.join(', ')} };\\n`);"], + ["await writeFile(join(dir, 'server.mjs'), `${shipped}\\nexport { nonexistentBinding };\\n`);"], + "the guard could not observe the numbers the service declares", + ("the guard could not observe the numbers the service declares",), + # The owners that could not be compared have to say so: an owner that held quietly is an + # owner nobody can tell apart from one that was checked and matched. + ("held vacuously",), + ), + ( + "the copy will not import and the owner compares the interval anyway", + [ + PROBE, + PROBE, + ], + [ + "await writeFile(join(dir, 'server.mjs'), `${shipped}\\nexport { ${names.join(', ')} };\\n`);", + "holds(createParsedRowsCache(1000).sweepIntervalMs === SHIPPED_SWEEP, true, BINDINGS !== null)", + ], + [ + "await writeFile(join(dir, 'server.mjs'), `${shipped}\\nexport { nonexistentBinding };\\n`);", + "holds(createParsedRowsCache(1000).sweepIntervalMs === SHIPPED_SWEEP, true, true)", + ], + "the guard accuses no rule it could not observe", + ( + "the guard accuses no rule it could not observe", + "the guard could not observe the numbers the service declares", + "the shipped default is the interval the service declares", + ), + (), + ), + ( + "the page's inline script does not run, so the owners below cannot read what they are about", + [CLIENT], + ["