diff --git a/.agents/notes/implemented/feature/2026-10-24-agent-transcript-images-and-task-notifications.md b/.agents/notes/implemented/feature/2026-10-24-agent-transcript-images-and-task-notifications.md new file mode 100644 index 000000000..bdda2e035 --- /dev/null +++ b/.agents/notes/implemented/feature/2026-10-24-agent-transcript-images-and-task-notifications.md @@ -0,0 +1,128 @@ +# Agent 笔记：对话中的图片缩略图、放大查看与后台任务通知 + +状态：已实现 + +## 先说结论 + +工具读取或截取图片时，对应的工具行（以及整组工具的标题行）右侧显示缩略图： +最多三张，其余用 `+N` 表示，悬停出大图预览，点击进入统一的放大查看器。用户消息里附带的图片也用同一套缩略图和查看器。 +Claude Code 发出的 `` 不再原样显示为用户气泡，而是一行状态文字。 +图片数据只在内存中以受限大小保存，原图由视图层按需异步解码。 + +## 问题 + +ACP（Agent Client Protocol，Agent 与客户端之间的协议）的工具结果可以携带 `image` 内容块， +但原先解析时只保留了类型名，图片本身被丢弃，用户看不到 Agent 读到的截图。 +用户发送的图片附件只以 `📎 路径` 文字出现，无法预览。Claude Code 的后台任务完成通知是协议标记， +却被当成用户输入显示。 + +## 决策 + +- `AgentToolDetails.images` 与 `AgentConversationMessage.images` 保存 `AgentImage`。内联图片解码后 + 不超过 10 MB、每条消息最多 24 张，超限直接丢弃，不进入会话状态。 + `AgentImage` 按内容指纹（SHA-256，硬件加速）或路径判等，转录行的相等比较不会去比较像素。 + 同一条消息、同一组工具里 id 相同的图片用 `uniquedByID` 去重，`ForEach` 和查看器翻页都要求 id 唯一。 + 内联数据必须能被 ImageIO 识别，否则不生成缩略图，工具输出里对应条目退回文字显示。 +- 只提供文件路径的 `read` 工具调用（例如读取 `.png`）直接显示该文件，且文件必须存在。其他类型的工具不会因为位置是图片就当作“查看”。 +- 一次更新如果没有图片，不会清掉之前已收到的图片，避免最终状态更新把缩略图抹掉。 +- 缩略图在视图层用 ImageIO 在后台线程按需生成并放入 `NSCache`（限制总像素成本和数量，文件缓存键含修改时间，行滚出后解码任务随之取消，解码失败显示占位图标）；历史回放只保存数据，不解码。 + 悬停预览复用 `workbenchHoverCard`，放大查看器是转录区域上的覆盖层，状态放在 + `AgentImageViewerState`，行只持有引用，打开或关闭不会重绘全部行。 +- 放大查看器仍是转录区域上的覆盖层，在其上补充捏合缩放（相对适应窗口 25%–800%）、拖动、双击、 + 百分比和 ⌘+/⌘−/⌘0，拖动范围限制在画面边缘内；←/→ 翻页只在查看器持有焦点时生效，不抢输入框的光标键；侧栏很窄时可放大查看。曾试过改用独立窗口复用编辑器的 `MediaViewerView`， + 因为偏离了既有交互而放弃。 +- 折叠的“思考过程”标题行也显示图片缩略图，否则默认折叠时用户看不到工具返回的图片。 +- 用户消息里的 `[@名字](file:///路径)`、裸 `file://` 地址、`📎 路径` 和 Claude Code 的 + `[Image: source: …]` 指向图片文件时显示为缩略图；`[Image: original …]` 尺寸说明丢弃。缩略图按引用在原文中的位置排序；超过 24 张的引用从文字中移除并显示 `+N`。 + 用户消息的解析结果按消息缓存，悬停和流式刷新不会重复扫描全文。 +- Claude Code 写进用户消息的特殊格式由 `AgentClaudeCodeMarkup` 识别：斜杠命令、本地命令输出、 + Shell 转义、中断提示、压缩续接摘要、技能展开和宿主上下文显示为状态行或折叠块； + ``、`` 等只给模型看的块只在消息开头或结尾时移除（Claude Code 就是加在那里）， + 夹在用户文字中间的同名标签原样保留。只有整条消息都是这些标记时才整体替换，用户自己输入的相似文字不会被吞掉。 + 格式清单来自对本机真实会话的标记名称统计，新增格式在这里补充。 +- `` 只有在整条用户消息完全由通知块组成时才解析，用户自己输入的相同文字不会被吞掉。 + 这是协议标记的呈现，不按 Agent 名称在宿主里分支。 + +### 后台任务通知并入后续过程 + +后台命令的完成通知会唤醒 Agent 再跑一轮。连续几次后台命令会让对话里反复出现“通知行 + 思考过程”。 +现在成功的通知（`AgentTranscriptItem.isQuietNotification`）不单独成行，而是进入随后一轮的折叠过程区，展开后按原顺序显示。 +失败、被终止的通知，以及后面没有任何思考或工具的通知，仍作为状态行保留，避免用户错过。 +Claude Code 为 shell 任务生成的英文摘要（`Background command "…" completed (exit code 0)`）按界面语言显示。 + +### 恢复会话时补全压缩前的历史 + +Claude Agent SDK 的 `getSessionMessages` 只返回当前 parent 链；压缩边界（`compact_boundary`）没有 parent， +因此适配器回放从最后一个边界开始，更早的记录虽然还在 `<配置目录>/projects/*/<会话 id>.jsonl`，却不会到达客户端， +对话里看起来像“前文丢了”。Rust 宿主（`claude_earlier_history.rs`）在 `session/load` 之前只读取该文件中最后一个边界之前的行， +转成与回放一致的 ACP 更新（用户、回复、思考、工具调用和结果），先于适配器的回放发出， +所以每次压缩的摘要也会在原位置显示为“对话已从摘要继续”。 + +这是对“宿主不读取转录文件”约定的有限例外，边界如下： + +- 只读，只打开 `projects/*/<会话 id>.jsonl`；账号目录里的登录信息和其他文件不触碰；会话 id 只接受字母、数字、`-`、`_`。 +- 配置目录按所选 Claude 账号解析（账号目录、`CLAUDE_CONFIG_DIR`、`~/.claude`），不写入任何文件，不缓存。 +- 超过 128 MiB 的记录文件不读取；最多发出 30000 条更新；工具输入输出每段截断到 4000 字符；子代理侧链和 `isMeta` 行跳过，图片不回放。 +- 文件格式由 Claude Code 拥有。解析失败或格式变化时只是不显示更早的历史，不影响适配器自身的回放。 +- Codex 的历史仍由其官方 App Server 提供，不受此例外影响。 + +不要这样做：不要把前文合并进适配器已回放的片段，也不要改写或重新压缩 Claude 自己的记录文件。 + +任务进行中排队发送的提示（含后台任务通知）在记录文件里只存成 `attachment` 行（`queued_command`）， +没有对应的用户行，所以前文转换把它当作用户消息回放，并用该行的 `uuid` 区分相邻消息。 + +### 兼容 Claude Desktop 开始的会话 + +Claude Desktop 给 Claude Code 挂了自己的 `ccd_turn` MCP 服务，模型在那里几乎不直接输出文字， +而是用 `mcp__ccd_turn__wrap_up` 工具交出每轮的总结（`recap`）和提问（`ask`），由 Desktop 渲染成卡片。 +在 Lithe 里恢复这类会话时，整段对话只剩折叠的工具调用，看起来像历史没有回放。 + +宿主（`claude_desktop.rs`）把 `wrap_up` 的工具调用改写成一条 Agent 回复（总结与提问各一段）， +并吞掉该调用后续的 `tool_call_update`；建议回复按钮不显示，因为用户点过的那条会作为下一条用户消息出现。 +改写同时作用于适配器的回放和上面的前文转换，Windows 共用同一宿主。Lithe 自己不提供 `ccd_turn` 工具， +所以只影响从 Desktop 带过来的历史。 + +`update_plan` 有整表（`items`）和增量（`done`/`start`/`advance`）两种写法。宿主按会话记住当前列表， +把每次调用改写成一条完整的 ACP `plan` 更新；增量调用之前没有整表时原样保留为工具调用。 +Swift 在加载历史期间把每份计划快照记成一条不带文字的 Agent 消息，转录里在两段折叠过程之间显示成清单， +导出 Markdown 时写成任务列表，不计入消息数。折叠的过程标题后面附上工具摘要（如“运行了 3 条命令”）， +没有文字回复的段落也能看出做了什么。 + +不要这样做：不要在 Swift 或 Windows 前端按工具名再做一份改写，也不要读取或模拟 Desktop 的 MCP 服务。 + +### 不要这样做 + +- 不要把图片 `Data` 放进 `Equatable` 的逐字节比较，也不要在主线程解码原图。 +- 不要为放大查看另起一套弹窗样式或使用 `.popover`；悬停预览与查看器共用本 Note 中的入口。 + +## 考虑过的备选方案 + +### 把图片写到缓存目录再引用路径 + +好处是内存占用更小。但应用层不能直接做文件系统写入，需要新增平台存储适配；在单张和总量都有上限的前提下， +先在内存中保留，需要时再评估。 + +### 保留“复制”按钮并在工具行内渲染 + +issue #1462 指出的多余复制按钮来自不属于最终回复的过程文字。#1464 已把这类文字收进默认折叠的过程区， +空白回复也不再生成行，因此本次没有再改动复制按钮的显示规则。 + +## 后果 + +- 收益：Agent 看过的图片可以直接在对话里确认；用户附件可点击放大。 +- 代价：长会话里的内联图片占用内存，最坏情况每条消息 24 × 10 MB。 +- 需要重新评估：出现明显内存压力，或需要 Windows 端对齐时。 + +## 验证 + +- `./scripts/test-macos.sh --filter AgentImageTests` +- `cargo test -p lithe-agent-host claude_earlier_history` +- `cargo test -p lithe-agent-host claude_desktop` +- `./scripts/verify-agent-notes.sh` + +## 适用范围 + +- `macos/Sources/LitheAgentConversationModule/Application/AgentImage.swift` +- `macos/Sources/Lithe/Views/Agent/AgentImageViews.swift` +- `macos/Sources/Lithe/Views/Agent/AgentToolGroupView.swift` +- `rust/lithe-agent-host/src/claude_earlier_history.rs` diff --git a/macos/Resources/en.lproj/Localizable.strings b/macos/Resources/en.lproj/Localizable.strings index 6e781d829..1fd605b17 100644 --- a/macos/Resources/en.lproj/Localizable.strings +++ b/macos/Resources/en.lproj/Localizable.strings @@ -374,8 +374,12 @@ "Back to all references" = "Back to all references"; "Back to the conversation" = "Back to the conversation"; "Background Services" = "Background Services"; +"Background command ended (exit code %@): %@" = "Background command ended (exit code %@): %@"; +"Background command finished: %@" = "Background command finished: %@"; "Background services" = "Background services"; "Background services started in Lithe are still running" = "Background services started in Lithe are still running"; +"Background task ended" = "Background task ended"; +"Background task finished" = "Background task finished"; "Balanced reasoning" = "Balanced reasoning"; "Base" = "Base"; "Base %@ → Target %@" = "Base %@ → Target %@"; @@ -807,6 +811,7 @@ "Conventional Commits" = "Conventional Commits"; "Conversation" = "Conversation"; "Conversation actions" = "Conversation actions"; +"Conversation continued from a summary" = "Conversation continued from a summary"; "Conversation history" = "Conversation history"; "Conversation history is unavailable" = "Conversation history is unavailable"; "Conversation tabs" = "Conversation tabs"; @@ -1185,6 +1190,7 @@ "End on next launch" = "End on next launch"; "End without asking" = "End without asking"; "English" = "English"; +"Enlarge image" = "Enlarge image"; "Enter API key" = "Enter API key"; "Enter a ChangeList name (1–100 characters)." = "Enter a ChangeList name (1–100 characters)."; "Enter a branch name" = "Enter a branch name"; @@ -1344,6 +1350,7 @@ "Find references to the selected symbol" = "Find references to the selected symbol"; "Finished" = "Finished"; "First %lld locations" = "First %lld locations"; +"Fit to Window" = "Fit to Window"; "Fix the failing checks below before installing." = "Fix the failing checks below before installing."; "Fixup" = "Fixup"; "Folder" = "Folder"; @@ -1632,6 +1639,7 @@ "Invalid shared instruction request." = "Invalid shared instruction request."; "Invert" = "Invert"; "It can read files, run commands and edit code after asking for permission." = "It can read files, run commands and edit code after asking for permission."; +"Item %lld of %lld" = "Item %lld of %lld"; "Its details and actions will appear here." = "Its details and actions will appear here."; "JDK Home" = "JDK Home"; "JDK home" = "JDK home"; @@ -2019,6 +2027,7 @@ "Next console expression" = "Next console expression"; "Next diff match" = "Next diff match"; "Next difference" = "Next difference"; +"Next image" = "Next image"; "Next match" = "Next match"; "Next match (Return)" = "Next match (Return)"; "Next run" = "Next run"; @@ -2406,6 +2415,7 @@ "Previous console expression" = "Previous console expression"; "Previous diff match" = "Previous diff match"; "Previous difference" = "Previous difference"; +"Previous image" = "Previous image"; "Previous match" = "Previous match"; "Previous match (Shift+Return)" = "Previous match (Shift+Return)"; "Previous session" = "Previous session"; @@ -3101,6 +3111,7 @@ "Skill files" = "Skill files"; "Skill imports cannot contain symbolic links or special files." = "Skill imports cannot contain symbolic links or special files."; "Skill instructions" = "Skill instructions"; +"Skill loaded: %@" = "Skill loaded: %@"; "Skill name (lowercase-with-hyphens)" = "Skill name (lowercase-with-hyphens)"; "Skill project" = "Skill project"; "Skill saved. Refresh or reconnect the Agent if it does not appear in the conversation." = "Skill saved. Refresh or reconnect the Agent if it does not appear in the conversation."; @@ -4207,9 +4218,13 @@ "native.backToAllReferences" = "Back to all references"; "native.backToTheConversation" = "Back to the conversation"; "native.backUpDatabaseAsSQL" = "Back Up Database as SQL…"; +"native.backgroundCommandEndedExitCodeArg1Arg2" = "Background command ended (exit code %1$@): %2$@"; +"native.backgroundCommandFinished" = "Background command finished: %1$@"; "native.backgroundServices" = "Background Services"; "native.backgroundServices.variant2" = "Background services"; "native.backgroundServicesStartedInLitheAreStillRunning" = "Background services started in Lithe are still running"; +"native.backgroundTaskEnded" = "Background task ended"; +"native.backgroundTaskFinished" = "Background task finished"; "native.balancedReasoning" = "Balanced reasoning"; "native.base" = "Base"; "native.base.variant2" = "Base: %1$@"; @@ -4645,6 +4660,7 @@ "native.conventionalCommits" = "Conventional Commits"; "native.conversation" = "Conversation"; "native.conversationActions" = "Conversation actions"; +"native.conversationContinuedFromASummary" = "Conversation continued from a summary"; "native.conversationHistory" = "Conversation history"; "native.conversationHistoryIsUnavailable" = "Conversation history is unavailable"; "native.conversationTabs" = "Conversation tabs"; @@ -5031,6 +5047,7 @@ "native.endThisTerminal" = "End This Terminal"; "native.endWithoutAsking" = "End without asking"; "native.english" = "English"; +"native.enlargeImage" = "Enlarge image"; "native.enterABranchName" = "Enter a branch name"; "native.enterABranchName.variant2" = "Enter a branch name."; "native.enterABranchNameBeforePublishing" = "Enter a branch name before publishing."; @@ -5194,6 +5211,7 @@ "native.finished" = "Finished"; "native.finishedWithoutCreating" = "%1$@ finished without creating %2$@."; "native.firstLldLocations" = "First %1$lld locations"; +"native.fitToWindow" = "Fit to Window"; "native.fixTheFailingChecksBelowBeforeInstalling" = "Fix the failing checks below before installing."; "native.fixup" = "Fixup"; "native.folder" = "Folder"; @@ -5587,6 +5605,7 @@ "native.isNotInstalledLitheInstallsItWithYourNpmIntoItsApplicationData" = "%1$@ is not installed. Lithe installs it with your npm into its application data."; "native.isVersionOrNewerIsRequired" = "%1$@ is version %2$@; %3$@ or newer is required."; "native.itCanReadFilesRunCommandsAndEditCodeAfterAskingForPermission" = "It can read files, run commands and edit code after asking for permission."; +"native.itemLldOfLld" = "Item %1$lld of %2$lld"; "native.itsDetailsAndActionsWillAppearHere" = "Its details and actions will appear here."; "native.japaneseCharacters" = "Japanese characters"; "native.javaApplication" = "Java Application"; @@ -6039,6 +6058,7 @@ "native.nextDiffMatch" = "Next diff match"; "native.nextDifference" = "Next difference"; "native.nextFile" = "Next File"; +"native.nextImage" = "Next image"; "native.nextMatch" = "Next Match"; "native.nextMatch.variant2" = "Next match"; "native.nextMatchReturn" = "Next match (Return)"; @@ -6427,6 +6447,7 @@ "native.previousDiffMatch" = "Previous diff match"; "native.previousDifference" = "Previous difference"; "native.previousFile" = "Previous File"; +"native.previousImage" = "Previous image"; "native.previousMatch" = "Previous Match"; "native.previousMatch.variant2" = "Previous match"; "native.previousMatchShiftReturn" = "Previous match (Shift+Return)"; @@ -7108,6 +7129,7 @@ "native.skillFiles" = "Skill files"; "native.skillImportsCannotContainSymbolicLinksOrSpecialFiles" = "Skill imports cannot contain symbolic links or special files."; "native.skillInstructions" = "Skill instructions"; +"native.skillLoaded" = "Skill loaded: %1$@"; "native.skillMdExceedsThe128KibLimit" = "SKILL.md exceeds the 128 KiB limit."; "native.skillMdFrontmatterIsNotClosed" = "SKILL.md frontmatter is not closed."; "native.skillMdHasInvalidYamlFrontmatter" = "SKILL.md has invalid YAML frontmatter."; diff --git a/macos/Resources/zh-Hans.lproj/Localizable.strings b/macos/Resources/zh-Hans.lproj/Localizable.strings index be55244ea..18910c056 100644 --- a/macos/Resources/zh-Hans.lproj/Localizable.strings +++ b/macos/Resources/zh-Hans.lproj/Localizable.strings @@ -374,8 +374,12 @@ "Back to all references" = "返回所有引用"; "Back to the conversation" = "返回对话"; "Background Services" = "后台服务"; +"Background command ended (exit code %@): %@" = "后台命令已结束（退出码 %@）：%@"; +"Background command finished: %@" = "后台命令已完成：%@"; "Background services" = "后台服务"; "Background services started in Lithe are still running" = "在 Lithe 中启动的后台服务仍在运行"; +"Background task ended" = "后台任务已结束"; +"Background task finished" = "后台任务已完成"; "Balanced reasoning" = "平衡思考深度与响应速度"; "Base" = "目标"; "Base %@ → Target %@" = "基准 %@ → 目标 %@"; @@ -807,6 +811,7 @@ "Conventional Commits" = "Conventional Commits 规范"; "Conversation" = "讨论"; "Conversation actions" = "会话操作"; +"Conversation continued from a summary" = "对话已从摘要继续"; "Conversation history" = "对话历史"; "Conversation history is unavailable" = "当前无法读取对话历史"; "Conversation tabs" = "会话标签"; @@ -1185,6 +1190,7 @@ "End on next launch" = "下次启动时结束"; "End without asking" = "直接结束"; "English" = "英文"; +"Enlarge image" = "放大查看图片"; "Enter API key" = "填写 API Key"; "Enter a ChangeList name (1–100 characters)." = "请输入更改列表名称（1–100 个字符）。"; "Enter a branch name" = "请输入分支名称"; @@ -1344,6 +1350,7 @@ "Find references to the selected symbol" = "查找所选符号的引用"; "Finished" = "已完成"; "First %lld locations" = "前 %lld 个位置"; +"Fit to Window" = "适应窗口"; "Fix the failing checks below before installing." = "请先解决下方未通过的检测项，再进行安装。"; "Fixup" = "合并并丢弃消息"; "Folder" = "分组"; @@ -1632,6 +1639,7 @@ "Invalid shared instruction request." = "共享指令请求无效。"; "Invert" = "反选"; "It can read files, run commands and edit code after asking for permission." = "它可以在征得你的许可后读取文件、运行命令和修改代码。"; +"Item %lld of %lld" = "第 %lld 项，共 %lld 项"; "Its details and actions will appear here." = "工作树详情和可用操作会显示在这里。"; "JDK Home" = "JDK 主目录"; "JDK home" = "JDK 主目录"; @@ -2019,6 +2027,7 @@ "Next console expression" = "下一条表达式"; "Next diff match" = "下一个 Diff 匹配项"; "Next difference" = "下一个差异"; +"Next image" = "下一张图片"; "Next match" = "下一个匹配"; "Next match (Return)" = "下一个匹配项（Return）"; "Next run" = "下次运行"; @@ -2406,6 +2415,7 @@ "Previous console expression" = "上一条表达式"; "Previous diff match" = "上一个 Diff 匹配项"; "Previous difference" = "上一个差异"; +"Previous image" = "上一张图片"; "Previous match" = "上一个匹配"; "Previous match (Shift+Return)" = "上一个匹配项（Shift+Return）"; "Previous session" = "上一次会话"; @@ -3101,6 +3111,7 @@ "Skill files" = "技能文件"; "Skill imports cannot contain symbolic links or special files." = "导入的技能不能包含符号链接或特殊文件。"; "Skill instructions" = "技能正文"; +"Skill loaded: %@" = "已加载技能：%@"; "Skill name (lowercase-with-hyphens)" = "技能名称（小写字母与连字符）"; "Skill project" = "当前项目"; "Skill saved. Refresh or reconnect the Agent if it does not appear in the conversation." = "技能已保存。如果对话中未显示，请刷新或重新连接 Agent。"; @@ -4207,9 +4218,13 @@ "native.backToAllReferences" = "返回所有引用"; "native.backToTheConversation" = "返回对话"; "native.backUpDatabaseAsSQL" = "将数据库备份为 SQL…"; +"native.backgroundCommandEndedExitCodeArg1Arg2" = "后台命令已结束（退出码 %1$@）：%2$@"; +"native.backgroundCommandFinished" = "后台命令已完成：%1$@"; "native.backgroundServices" = "后台服务"; "native.backgroundServices.variant2" = "后台服务"; "native.backgroundServicesStartedInLitheAreStillRunning" = "在 Lithe 中启动的后台服务仍在运行"; +"native.backgroundTaskEnded" = "后台任务已结束"; +"native.backgroundTaskFinished" = "后台任务已完成"; "native.balancedReasoning" = "平衡思考深度与响应速度"; "native.base" = "目标"; "native.base.variant2" = "基点：%1$@"; @@ -4645,6 +4660,7 @@ "native.conventionalCommits" = "Conventional Commits 规范"; "native.conversation" = "讨论"; "native.conversationActions" = "会话操作"; +"native.conversationContinuedFromASummary" = "对话已从摘要继续"; "native.conversationHistory" = "对话历史"; "native.conversationHistoryIsUnavailable" = "当前无法读取对话历史"; "native.conversationTabs" = "会话标签"; @@ -5031,6 +5047,7 @@ "native.endThisTerminal" = "结束此终端"; "native.endWithoutAsking" = "直接结束"; "native.english" = "英文"; +"native.enlargeImage" = "放大查看图片"; "native.enterABranchName" = "请输入分支名称"; "native.enterABranchName.variant2" = "请输入分支名称。"; "native.enterABranchNameBeforePublishing" = "请输入分支名称后再发布。"; @@ -5194,6 +5211,7 @@ "native.finished" = "已完成"; "native.finishedWithoutCreating" = "%1$@ 已结束，但没有生成 %2$@。"; "native.firstLldLocations" = "前 %1$lld 个位置"; +"native.fitToWindow" = "适应窗口"; "native.fixTheFailingChecksBelowBeforeInstalling" = "请先解决下方未通过的检测项，再进行安装。"; "native.fixup" = "合并并丢弃消息"; "native.folder" = "分组"; @@ -5587,6 +5605,7 @@ "native.isNotInstalledLitheInstallsItWithYourNpmIntoItsApplicationData" = "%1$@ 尚未安装。Lithe 会用你的 npm 把它安装到应用数据目录。"; "native.isVersionOrNewerIsRequired" = "%1$@ 的版本是 %2$@，需要 %3$@ 或更高版本。"; "native.itCanReadFilesRunCommandsAndEditCodeAfterAskingForPermission" = "它可以在征得你的许可后读取文件、运行命令和修改代码。"; +"native.itemLldOfLld" = "第 %1$lld 项，共 %2$lld 项"; "native.itsDetailsAndActionsWillAppearHere" = "工作树详情和可用操作会显示在这里。"; "native.japaneseCharacters" = "日文字符"; "native.javaApplication" = "Java 应用"; @@ -6039,6 +6058,7 @@ "native.nextDiffMatch" = "下一个 Diff 匹配项"; "native.nextDifference" = "下一个差异"; "native.nextFile" = "下一个文件"; +"native.nextImage" = "下一张图片"; "native.nextMatch" = "下一个匹配项"; "native.nextMatch.variant2" = "下一个匹配"; "native.nextMatchReturn" = "下一个匹配项（Return）"; @@ -6427,6 +6447,7 @@ "native.previousDiffMatch" = "上一个 Diff 匹配项"; "native.previousDifference" = "上一个差异"; "native.previousFile" = "上一个文件"; +"native.previousImage" = "上一张图片"; "native.previousMatch" = "上一个匹配项"; "native.previousMatch.variant2" = "上一个匹配"; "native.previousMatchShiftReturn" = "上一个匹配项（Shift+Return）"; @@ -7108,6 +7129,7 @@ "native.skillFiles" = "技能文件"; "native.skillImportsCannotContainSymbolicLinksOrSpecialFiles" = "导入的技能不能包含符号链接或特殊文件。"; "native.skillInstructions" = "技能正文"; +"native.skillLoaded" = "已加载技能：%1$@"; "native.skillMdExceedsThe128KibLimit" = "SKILL.md 超过了 128 KiB 的大小限制。"; "native.skillMdFrontmatterIsNotClosed" = "SKILL.md 的文件头缺少结束标记。"; "native.skillMdHasInvalidYamlFrontmatter" = "SKILL.md 的 YAML 文件头格式不正确。"; diff --git a/macos/Sources/Lithe/Views/Agent/AgentHistoryView.swift b/macos/Sources/Lithe/Views/Agent/AgentHistoryView.swift index 53029102a..9eae1185a 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentHistoryView.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentHistoryView.swift @@ -210,7 +210,7 @@ enum AgentHistoryPresentation { var hasTranscript = false for message in conversation.messages where message.subagentSessionID == nil || message.role == .tool { hasTranscript = true - if message.role == .user || message.role == .agent { count += 1 } + if message.role == .user || (message.role == .agent && message.plan == nil) { count += 1 } } return conversation.isAttached || hasTranscript ? count : nil } diff --git a/macos/Sources/Lithe/Views/Agent/AgentImageViews.swift b/macos/Sources/Lithe/Views/Agent/AgentImageViews.swift new file mode 100644 index 000000000..7ec1eee4a --- /dev/null +++ b/macos/Sources/Lithe/Views/Agent/AgentImageViews.swift @@ -0,0 +1,409 @@ +import AppKit +import ImageIO +import SwiftUI +import LitheAgentConversationModule + +/// Images of one tool group or message, opened at `index` in the enlarged viewer. +struct AgentImageGallery: Identifiable, Equatable { + let id = UUID() + let images: [AgentImage] + var index: Int +} + +/// Owns the open viewer. Rows hold only this reference, so opening or closing +/// the viewer never re-evaluates the transcript rows themselves. +@MainActor +final class AgentImageViewerState: ObservableObject { + @Published var gallery: AgentImageGallery? + + func open(_ images: [AgentImage], at index: Int) { + guard images.indices.contains(index) else { return } + gallery = AgentImageGallery(images: images, index: index) + } +} + +private struct AgentImageViewerStateKey: EnvironmentKey { + static let defaultValue: AgentImageViewerState? = nil +} + +extension EnvironmentValues { + var agentImageViewer: AgentImageViewerState? { + get { self[AgentImageViewerStateKey.self] } + set { self[AgentImageViewerStateKey.self] = newValue } + } +} + +/// Decodes bounded renditions off the main thread. Inline pictures can be several +/// megabytes, so rows ask for a thumbnail-sized decode and share it through a cache. +enum AgentImageLoader { + private static let cache: NSCache = { + let cache = NSCache() + // Large renditions are tens of megabytes decoded, so bound by pixels, not just count. + cache.totalCostLimit = 192 * 1024 * 1024 + cache.countLimit = 400 + return cache + }() + + static func cachedImage(for image: AgentImage, maxPixel: Int) -> NSImage? { + cache.object(forKey: key(image, maxPixel)) + } + + /// Nil when the picture cannot be decoded, or when the calling task was cancelled. + static func image(for image: AgentImage, maxPixel: Int) async -> NSImage? { + if let cached = cachedImage(for: image, maxPixel: maxPixel) { return cached } + let source = image.source + let cacheKey = key(image, maxPixel) + let decode = Task.detached(priority: .utility) { () -> CGImage? in + if Task.isCancelled { return nil } + let imageSource: CGImageSource? + switch source { + case .data(let data): imageSource = CGImageSourceCreateWithData(data as CFData, nil) + case .file(let path): imageSource = CGImageSourceCreateWithURL(URL(fileURLWithPath: path) as CFURL, nil) + } + guard let imageSource, !Task.isCancelled else { return nil } + return CGImageSourceCreateThumbnailAtIndex(imageSource, 0, [ + kCGImageSourceCreateThumbnailFromImageAlways: true, + kCGImageSourceCreateThumbnailWithTransform: true, + kCGImageSourceThumbnailMaxPixelSize: maxPixel, + ] as CFDictionary) + } + // A row scrolled away cancels its task; stop the decode instead of finishing unseen work. + let rendition = await withTaskCancellationHandler { await decode.value } onCancel: { decode.cancel() } + guard let rendition else { return nil } + let result = NSImage(cgImage: rendition, size: NSSize(width: rendition.width, height: rendition.height)) + cache.setObject(result, forKey: cacheKey, cost: rendition.width * rendition.height * 4) + return result + } + + /// Files carry their modification time, so a screenshot rewritten at the same path is decoded again. + private static func key(_ image: AgentImage, _ maxPixel: Int) -> NSString { + var stamp = "" + if case .file(let path) = image.source, + let date = try? URL(fileURLWithPath: path).resourceValues(forKeys: [.contentModificationDateKey]).contentModificationDate { + stamp = "@\(date.timeIntervalSinceReferenceDate)" + } + return "\(image.id)\(stamp)@\(maxPixel)" as NSString + } +} + +/// A picture that fills its frame, or fits inside it, once decoded. +struct AgentImageThumbnail: View { + let image: AgentImage + var maxPixel = 160 + var fills = true + @State private var rendition: NSImage? + @State private var failed = false + + var body: some View { + ZStack { + if let rendition { + Image(nsImage: rendition).resizable().aspectRatio(contentMode: fills ? .fill : .fit) + } else { + Rectangle().fill(LitheTheme.raised) + if failed { + Image(systemName: "photo").foregroundStyle(LitheTheme.secondaryText).imageScale(.small) + } + } + } + .task(id: image.id) { + failed = false + rendition = AgentImageLoader.cachedImage(for: image, maxPixel: maxPixel) + if rendition == nil { + let decoded = await AgentImageLoader.image(for: image, maxPixel: maxPixel) + if Task.isCancelled { return } + rendition = decoded + failed = decoded == nil + } + } + } +} + +/// Newest pictures beside a tool row or group, `+N` for the rest. Hovering previews +/// one; clicking opens the whole set in the viewer. +struct AgentImageStrip: View { + @Environment(\.agentImageViewer) private var viewer + let images: [AgentImage] + static let visibleLimit = 3 + static let thumbnailSize = CGSize(width: 36, height: 26) + + var body: some View { + let first = max(images.count - Self.visibleLimit, 0) + HStack(spacing: 6) { + if first > 0 { + Text(verbatim: "+\(first)") + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(LitheTheme.secondaryText) + } + ForEach(Array(images.enumerated()).suffix(Self.visibleLimit), id: \.element.id) { index, image in + AgentImageThumbnail(image: image) + .frame(width: Self.thumbnailSize.width, height: Self.thumbnailSize.height) + .clipShape(RoundedRectangle(cornerRadius: 5)) + .overlay(RoundedRectangle(cornerRadius: 5).stroke(LitheTheme.panelBorder, lineWidth: 1)) + .contentShape(Rectangle()) + .lithePointer() + .onTapGesture { viewer?.open(images, at: index) } + .workbenchHoverCard(placement: .above, maximumWidth: AgentImagePreviewCard.width) { + AgentImagePreviewCard(image: image, position: index + 1, count: images.count) + } + .accessibilityElement() + .accessibilityAddTraits(.isButton) + .accessibilityLabel(Text(verbatim: image.title ?? "Image")) + .accessibilityHint(Text("Enlarge image")) + } + } + .fixedSize() + .accessibilityIdentifier("agent-image-strip") + } +} + +/// Non-interactive hover card with a larger rendition and its caption. +private struct AgentImagePreviewCard: View { + @Environment(\.locale) private var locale + let image: AgentImage + let position: Int + let count: Int + static let width: CGFloat = 340 + + var body: some View { + VStack(alignment: .leading, spacing: 0) { + AgentImageThumbnail(image: image, maxPixel: 960, fills: false) + .frame(maxWidth: Self.width, maxHeight: 220) + .frame(minHeight: 60) + HStack(spacing: 8) { + Text(verbatim: image.title ?? "") + .lineLimit(1).truncationMode(.middle) + Spacer(minLength: 8) + if count > 1 { + Text(String(format: AgentLocalization.string("Item %lld of %lld", locale: locale), position, count)) + .foregroundStyle(LitheTheme.secondaryText) + } + } + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(LitheTheme.HoverTooltip.foreground) + .padding(.horizontal, 12) + .padding(.vertical, 9) + } + .frame(width: Self.width) + .litheHoverTooltipSurface() + } +} + +/// Attachments of a user message, larger than tool thumbnails. +struct AgentUserImages: View { + @Environment(\.agentImageViewer) private var viewer + let images: [AgentImage] + /// References past the per-message cap, summarised as `+N` like the tool strip. + var hiddenCount = 0 + + var body: some View { + HStack(spacing: 6) { + ForEach(Array(images.enumerated()), id: \.element.id) { index, image in + AgentImageThumbnail(image: image, maxPixel: 320) + .frame(width: 84, height: 64) + .clipShape(RoundedRectangle(cornerRadius: 7)) + .overlay(RoundedRectangle(cornerRadius: 7).stroke(LitheTheme.panelBorder, lineWidth: 1)) + .contentShape(Rectangle()) + .lithePointer() + .onTapGesture { viewer?.open(images, at: index) } + .accessibilityElement() + .accessibilityAddTraits(.isButton) + .accessibilityLabel(Text(verbatim: image.title ?? "Image")) + .accessibilityHint(Text("Enlarge image")) + } + if hiddenCount > 0 { + Text(verbatim: "+\(hiddenCount)") + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(LitheTheme.secondaryText) + } + } + .accessibilityIdentifier("agent-user-images") + } +} + +/// Full-pane viewer: dimmed backdrop, image fitted to the pane, arrows and Esc. +struct AgentImageViewer: View { + @Environment(\.locale) private var locale + @ObservedObject var state: AgentImageViewerState + let gallery: AgentImageGallery + /// Relative to the fitted size, so 100% always shows the whole picture. + @State private var zoom: CGFloat = 1 + @State private var pinchBase: CGFloat = 1 + @State private var pan: CGSize = .zero + @State private var panBase: CGSize = .zero + @State private var viewport: CGSize = .zero + /// Arrow keys page only while the viewer holds focus, so a caret in the composer keeps them. + @FocusState private var hasFocus: Bool + private static let zoomRange: ClosedRange = 0.25...8 + + var body: some View { + let image = gallery.images[gallery.index] + ZStack { + Color.black.opacity(0.62) + .contentShape(Rectangle()) + .onTapGesture { state.gallery = nil } + VStack(spacing: 10) { + AgentImageThumbnail(image: image, maxPixel: 2400, fills: false) + .scaleEffect(zoom) + .offset(pan) + .frame(maxWidth: .infinity, maxHeight: .infinity) + .background(GeometryReader { proxy in + Color.clear.onAppear { viewport = proxy.size } + .onChange(of: proxy.size) { viewport = $0 } + }) + .clipped() + .shadow(color: .black.opacity(0.35), radius: 12) + .contentShape(Rectangle()) + .gesture(MagnificationGesture() + .onChanged { setZoom(pinchBase * $0) } + .onEnded { _ in pinchBase = zoom }) + .simultaneousGesture(DragGesture() + .onChanged { if zoom > 1 { pan = clamped(CGSize(width: panBase.width + $0.translation.width, + height: panBase.height + $0.translation.height)) } } + .onEnded { _ in panBase = pan }) + .onTapGesture(count: 2) { withAnimation(.none) { zoom > 1 ? resetZoom() : setZoom(2) } } + .onChange(of: gallery.index) { _ in resetZoom() } + HStack(spacing: 8) { + Text(verbatim: image.title ?? "") + .lineLimit(1).truncationMode(.middle) + if gallery.images.count > 1 { + Text(String(format: AgentLocalization.string("Item %lld of %lld", locale: locale), + gallery.index + 1, gallery.images.count)) + .foregroundStyle(.white.opacity(0.7)) + } + } + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(.white) + zoomControls + } + .padding(.horizontal, 56) + .padding(.vertical, 40) + HStack { + if gallery.images.count > 1 { + step(-1, systemImage: "chevron.left", label: "Previous image") + } + Spacer() + if gallery.images.count > 1 { + step(1, systemImage: "chevron.right", label: "Next image") + } + } + .padding(.horizontal, 12) + VStack { + HStack { + Spacer() + Button { state.gallery = nil } label: { + Image(systemName: "xmark").font(LitheTheme.uiFont(size: 13, weight: .semibold)) + .frame(width: 28, height: 28) + .background(.white.opacity(0.16), in: Circle()) + .contentShape(Circle()) + } + .buttonStyle(.litheNoPress) + .foregroundStyle(.white) + .keyboardShortcut(.cancelAction) + .lithePointer() + .help("Close") + .accessibilityLabel("Close") + } + Spacer() + } + .padding(12) + } + .modifier(AgentImagePagingKeys(hasFocus: $hasFocus, canPage: gallery.images.count > 1, page: { page($0) })) + .accessibilityElement(children: .contain) + .accessibilityIdentifier("agent-image-viewer") + } + + /// Keeps at least the picture's edge in view: the scaled picture may overhang the pane by half its extra size. + private func clamped(_ offset: CGSize) -> CGSize { + let limitX = viewport.width * (zoom - 1) / 2, limitY = viewport.height * (zoom - 1) / 2 + return CGSize(width: min(max(offset.width, -limitX), limitX), height: min(max(offset.height, -limitY), limitY)) + } + + private func page(_ delta: Int) { + let count = gallery.images.count + guard count > 1 else { return } + state.gallery?.index = (gallery.index + delta + count) % count + } + + private var zoomControls: some View { + HStack(spacing: 6) { + zoomButton("minus", help: "Zoom Out", key: "-") { setZoom(zoom / 1.25) } + Button { resetZoom() } label: { + Text(verbatim: "\(Int((zoom * 100).rounded()))%") + .font(LitheTheme.uiFont(size: 12, weight: .medium)).monospacedDigit() + .frame(minWidth: 48, minHeight: 24) + .background(.white.opacity(0.16), in: Capsule()) + .contentShape(Capsule()) + } + .buttonStyle(.litheNoPress) + .foregroundStyle(.white) + .keyboardShortcut("0", modifiers: .command) + .lithePointer() + .help("Fit to Window") + zoomButton("plus", help: "Zoom In", key: "=") { setZoom(zoom * 1.25) } + } + } + + private func zoomButton(_ systemImage: String, help: LocalizedStringKey, key: KeyEquivalent, + action: @escaping () -> Void) -> some View { + Button(action: action) { + Image(systemName: systemImage).font(LitheTheme.uiFont(size: 11, weight: .semibold)) + .frame(width: 24, height: 24) + .background(.white.opacity(0.16), in: Circle()) + .contentShape(Circle()) + } + .buttonStyle(.litheNoPress) + .foregroundStyle(.white) + .keyboardShortcut(key, modifiers: .command) + .lithePointer() + .help(help) + .accessibilityLabel(help) + } + + private func setZoom(_ value: CGFloat) { + zoom = min(max(value, Self.zoomRange.lowerBound), Self.zoomRange.upperBound) + pinchBase = zoom + if zoom <= 1 { pan = .zero; panBase = .zero } else { pan = clamped(pan); panBase = pan } + } + + private func resetZoom() { + zoom = 1; pinchBase = 1; pan = .zero; panBase = .zero + } + + private func step(_ delta: Int, systemImage: String, label: LocalizedStringKey) -> some View { + Button { + page(delta) + } label: { + Image(systemName: systemImage).font(LitheTheme.uiFont(size: 14, weight: .semibold)) + .frame(width: 32, height: 32) + .background(.white.opacity(0.16), in: Circle()) + .contentShape(Circle()) + } + .buttonStyle(.litheNoPress) + .foregroundStyle(.white) + .lithePointer() + .help(label) + .accessibilityLabel(label) + } +} + +/// ←/→ page the viewer only while it holds focus, so the composer keeps its caret keys. +/// Key handling needs macOS 14; macOS 13 pages with the on-screen arrows. +private struct AgentImagePagingKeys: ViewModifier { + var hasFocus: FocusState.Binding + let canPage: Bool + let page: (Int) -> Void + + func body(content: Content) -> some View { + if #available(macOS 14.0, *) { + content + .focusable() + .focusEffectDisabled() + .focused(hasFocus) + .onAppear { hasFocus.wrappedValue = true } + .onKeyPress(.leftArrow) { canPage ? { page(-1); return .handled }() : .ignored } + .onKeyPress(.rightArrow) { canPage ? { page(1); return .handled }() : .ignored } + } else { + content + } + } +} diff --git a/macos/Sources/Lithe/Views/Agent/AgentProcessGroupView.swift b/macos/Sources/Lithe/Views/Agent/AgentProcessGroupView.swift index a8fbaeec3..815c27849 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentProcessGroupView.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentProcessGroupView.swift @@ -15,31 +15,62 @@ struct AgentProcessGroupView: View { expansion.isExpanded(isStreaming: isStreaming, isSearching: !searchText.isEmpty) } + private var headerButton: some View { + Button { expansion.toggle(isStreaming: isStreaming, isSearching: !searchText.isEmpty) } label: { + HStack(spacing: 5) { + if isStreaming { ProgressView().controlSize(.mini) } + Text("Thinking process").lineLimit(1) + Image(systemName: "chevron.down") + .font(LitheTheme.uiFont(size: 9, weight: .semibold)) + .rotationEffect(.degrees(isExpanded ? 0 : -90)) + // Like Claude Desktop's "Ran 3 commands": a collapsed process still says what it did. + if !isExpanded && !toolSummary.isEmpty { + Text(verbatim: toolSummary) + .lineLimit(1) + .truncationMode(.tail) + .foregroundStyle(LitheTheme.tertiaryText) + } + } + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(LitheTheme.secondaryText) + .padding(.vertical, 2) + .contentShape(Rectangle()) + } + .buttonStyle(.litheNoPress) + .lithePointer() + .help(isExpanded ? "Hide thinking" : "Show thinking") + .accessibilityValue(isExpanded ? AgentLocalization.string("Expanded", locale: locale) : AgentLocalization.string("Collapsed", locale: locale)) + .accessibilityIdentifier("agent-process-group-header") + } + + private var toolSummary: String { + AgentToolGroupPresentation.summary(messages.filter { $0.role == .tool }, locale: locale) + } + + private var images: [AgentImage] { + messages.flatMap(\.toolDetails.images).uniquedByID + } + var body: some View { VStack(alignment: .leading, spacing: 8) { - Button { expansion.toggle(isStreaming: isStreaming, isSearching: !searchText.isEmpty) } label: { - HStack(spacing: 5) { - if isStreaming { ProgressView().controlSize(.mini) } - Text("Thinking process").lineLimit(1) - Image(systemName: "chevron.down") - .font(LitheTheme.uiFont(size: 9, weight: .semibold)) - .rotationEffect(.degrees(isExpanded ? 0 : -90)) - } - .font(LitheTheme.uiFont(size: 12.5)) - .foregroundStyle(LitheTheme.secondaryText) - .padding(.vertical, 2) - .contentShape(Rectangle()) + HStack(spacing: 8) { + headerButton + Spacer(minLength: 8) + // The process is collapsed by default; pictures the tools returned stay visible beside it; expanded, each tool group shows its own. + if !isExpanded && !images.isEmpty { AgentImageStrip(images: images) } } - .buttonStyle(.litheNoPress) - .lithePointer() - .help(isExpanded ? "Hide thinking" : "Show thinking") - .accessibilityValue(isExpanded ? AgentLocalization.string("Expanded", locale: locale) : AgentLocalization.string("Collapsed", locale: locale)) - .accessibilityIdentifier("agent-process-group-header") if isExpanded { VStack(alignment: .leading, spacing: 8) { ForEach(AgentTranscriptItem.processContents(messages).filter { $0.matches(searchText) }) { item in switch item { + case .message(let message) where message.role == .user: + // Background-task notices that woke the Agent for this turn. + VStack(alignment: .leading, spacing: 4) { + ForEach(Array((AgentClaudeCodeMarkup.parse(message.text) ?? []).enumerated()), id: \.offset) { _, entry in + AgentClaudeCodeEntryRow(entry: entry) + } + } case .message(let message): AgentMarkdownMessage(text: message.text.trimmingCharacters(in: .whitespacesAndNewlines), size: 12) .equatable() diff --git a/macos/Sources/Lithe/Views/Agent/AgentSessionGuidanceViews.swift b/macos/Sources/Lithe/Views/Agent/AgentSessionGuidanceViews.swift index 8cafdb5c5..0c84d1b98 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentSessionGuidanceViews.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentSessionGuidanceViews.swift @@ -103,12 +103,15 @@ struct AgentPlanView: View { let plan: AgentPlan let isResponding: Bool let embedded: Bool + /// Pinned plans align with the composer; transcript entries align with messages. + let horizontalInset: CGFloat @State private var expanded = false - init(plan: AgentPlan, isResponding: Bool, expanded: Bool = false, embedded: Bool = false) { + init(plan: AgentPlan, isResponding: Bool, expanded: Bool = false, embedded: Bool = false, horizontalInset: CGFloat = 18) { self.plan = plan self.isResponding = isResponding self.embedded = embedded + self.horizontalInset = horizontalInset _expanded = State(initialValue: expanded) } @@ -162,7 +165,7 @@ struct AgentPlanView: View { .frame(maxWidth: .infinity, alignment: .leading) .background(embedded ? Color.clear : AgentPanelStyle.header, in: RoundedRectangle(cornerRadius: 5)) .overlay(RoundedRectangle(cornerRadius: 5).stroke(embedded ? Color.clear : AgentPanelStyle.border, lineWidth: 1)) - .padding(.horizontal, embedded ? 0 : 18) + .padding(.horizontal, embedded ? 0 : horizontalInset) .padding(.bottom, embedded ? 0 : 4) } diff --git a/macos/Sources/Lithe/Views/Agent/AgentToolGroupView.swift b/macos/Sources/Lithe/Views/Agent/AgentToolGroupView.swift index d1e13a9fa..0285c7b04 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentToolGroupView.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentToolGroupView.swift @@ -73,6 +73,7 @@ struct AgentToolGroupView: View { @State private var showsAllRows = false @State private var expandedToolID: String? + private var groupImages: [AgentImage] { messages.flatMap(\.toolDetails.images).uniquedByID } private var failedCount: Int { messages.filter { $0.toolStatus == .failed }.count } private var isRunning: Bool { messages.contains { $0.toolStatus == .inProgress || $0.toolStatus == .pending } } private var hasExpandedDetails: Bool { @@ -107,7 +108,15 @@ struct AgentToolGroupView: View { private var header: some View { let summary = AgentToolGroupPresentation.summary(messages, locale: locale) - return Button { expanded.toggle() } label: { + return HStack(spacing: 8) { + headerButton(summary: summary) + Spacer(minLength: 8) + if !groupImages.isEmpty { AgentImageStrip(images: groupImages) } + } + } + + private func headerButton(summary: String) -> some View { + Button { expanded.toggle() } label: { HStack(spacing: 5) { if isRunning { ProgressView().controlSize(.mini) } Text(verbatim: summary).lineLimit(1) @@ -172,6 +181,7 @@ struct AgentToolGroupView: View { .lineLimit(1) .truncationMode(.middle) .frame(maxWidth: .infinity, alignment: .leading) + if !message.toolDetails.images.isEmpty { AgentImageStrip(images: message.toolDetails.images) } status(message.toolStatus) if hasDetails { Image(systemName: "chevron.right") diff --git a/macos/Sources/Lithe/Views/Agent/AgentTranscriptNavigation.swift b/macos/Sources/Lithe/Views/Agent/AgentTranscriptNavigation.swift index 0f5772b21..cb0548cb9 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentTranscriptNavigation.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentTranscriptNavigation.swift @@ -16,7 +16,8 @@ struct AgentTranscriptNavigationContent { init(messages: [AgentConversationMessage]) { for message in messages { - if message.role == .user { + // Claude Code's own markup (commands, notifications, recaps) is not a prompt the user wrote. + if message.role == .user, AgentClaudeCodeMarkup.parse(message.text) == nil { entries.append(Entry(id: message.id, prompt: String(message.text.prefix(320)))) } guard let turnID = entries.last?.id else { continue } diff --git a/macos/Sources/Lithe/Views/Agent/AgentTranscriptView.swift b/macos/Sources/Lithe/Views/Agent/AgentTranscriptView.swift index da48729b1..bb8aa498f 100644 --- a/macos/Sources/Lithe/Views/Agent/AgentTranscriptView.swift +++ b/macos/Sources/Lithe/Views/Agent/AgentTranscriptView.swift @@ -38,7 +38,9 @@ enum AgentTranscriptItem: Identifiable { func matches(_ searchText: String) -> Bool { guard !searchText.isEmpty else { return true } switch self { - case .message(let message): return message.text.localizedStandardContains(searchText) + case .message(let message): + return message.text.localizedStandardContains(searchText) + || message.plan?.entries.contains { $0.content.localizedStandardContains(searchText) } == true case .processGroup(let messages): return messages.contains { $0.role == .tool ? Self.toolMatches($0, searchText) : $0.text.localizedStandardContains(searchText) } @@ -67,7 +69,13 @@ enum AgentTranscriptItem: Identifiable { // ACP has no final/commentary role. Only text after the last thought // or tool is presented as the reply; earlier prose is process narration. if let lastActivity = pending.lastIndex(where: { $0.role != .agent }) { - items.append(.processGroup(Array(pending[...lastActivity]))) + let process = Array(pending[...lastActivity]) + // Notices that triggered no thinking or tool use have no process to join. + if process.allSatisfy(Self.isQuietNotification) { + items += process.map(Self.message) + } else { + items.append(.processGroup(process)) + } items += pending.dropFirst(lastActivity + 1).map(Self.message) } else { items += pending.map(Self.message) @@ -75,7 +83,21 @@ enum AgentTranscriptItem: Identifiable { pending.removeAll(keepingCapacity: true) } for message in messages { - if message.role == .user { + if message.role == .user, case .markup(let entries) = AgentUserMessagePresentation.of(message), entries.isEmpty { + // Model-only reminders and caveats are not conversation. + continue + } + if message.plan != nil { + // A replayed plan snapshot is what the Agent reported at that point; + // it stays on the transcript like Claude Desktop's checklist. + flush() + items.append(.message(message)) + } else if Self.isQuietNotification(message) { + // A finished background task wakes the Agent for a follow-up turn. + // The notice joins that turn's collapsed process instead of + // interrupting the transcript between two replies. + pending.append(message) + } else if message.role == .user { flush() items.append(.message(message)) } else if message.role == .tool || !message.text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { @@ -91,6 +113,16 @@ enum AgentTranscriptItem: Identifiable { return items } + /// A successful background-task notice: protocol markup that needs no attention. + /// Failures stay on the transcript where the user can see them. + static func isQuietNotification(_ message: AgentConversationMessage) -> Bool { + guard message.role == .user, case .markup(let entries) = AgentUserMessagePresentation.of(message), !entries.isEmpty else { return false } + return entries.allSatisfy { + if case .taskNotification(let notification) = $0 { return !notification.isFailure } + return false + } + } + /// Within an expanded process, keep prose in order and combine adjacent tools. static func processContents(_ messages: [AgentConversationMessage]) -> [Self] { var items: [Self] = [] @@ -163,6 +195,8 @@ struct AgentTranscriptView: View { var isEditorLayout = false @State private var navigation = AgentTranscriptNavigationState() @State private var showsAgentPicker = false + // A reference the rows share; only the overlay observes it, so opening an image does not re-evaluate this view. + @State private var imageViewer = AgentImageViewerState() // A filtered-out row can be destroyed. Keep its preference in the owning // transcript view, isolated by session and message, until the actual data goes away. @State private var thoughtExpansions: [String: [String: AgentThoughtExpansion]] = [:] @@ -309,6 +343,8 @@ struct AgentTranscriptView: View { .id(sessionID) } } + .environment(\.agentImageViewer, imageViewer) + .overlay { AgentImageViewerOverlay(state: imageViewer) } .onChange(of: feature.openSessionIDs) { sessionIDs in thoughtExpansions = thoughtExpansions.filter { sessionIDs.contains($0.key) } windowLimits = windowLimits.filter { sessionIDs.contains($0.key) } @@ -332,6 +368,17 @@ struct AgentTranscriptView: View { } } +/// Only this overlay observes the viewer, so opening an image leaves the rows untouched. +private struct AgentImageViewerOverlay: View { + @ObservedObject var state: AgentImageViewerState + + var body: some View { + if let gallery = state.gallery { + AgentImageViewer(state: state, gallery: gallery) + } + } +} + /// Transcript rows as a value of the conversation data. The owning view is /// re-evaluated for every published model change (quota, sessions, statistics); /// comparing these inputs keeps scrolling from re-diffing every row each time. @@ -373,6 +420,8 @@ private struct AgentTranscriptRows: View, Equatable { ForEach(transcript) { item in Group { switch item { + case .message(let message) where message.plan != nil: + AgentPlanView(plan: message.plan!, isResponding: false, horizontalInset: 0) case .message(let message): AgentMessageRow( message: message, @@ -529,33 +578,16 @@ private struct AgentMessageRow: View { var body: some View { switch message.role { case .user: - HStack(alignment: .bottom, spacing: 4) { - Spacer(minLength: 40) - AgentCopyButton(text: message.text, help: "Copy message") - .opacity(isHovered ? 1 : 0) - .allowsHitTesting(isHovered) - VStack(alignment: .trailing, spacing: 4) { - Text(message.text) - if let delivery = message.delivery { - Group { - switch delivery { - case .sending: Text("Sending…") - case .received: Text("Agent received") - case .failed(let reason): Text(AgentLocalization.message(reason, locale: locale)).foregroundStyle(LitheTheme.error) - } - } - .font(LitheTheme.uiFont(size: 10)) - .foregroundStyle(AgentPanelStyle.secondary) + if case .markup(let entries) = AgentUserMessagePresentation.of(message) { + VStack(alignment: .leading, spacing: 4) { + ForEach(Array(entries.enumerated()), id: \.offset) { _, entry in + AgentClaudeCodeEntryRow(entry: entry) } } - .font(LitheTheme.uiFont(size: isEditorLayout ? 14 : 13)) - .textSelection(.enabled) - .padding(.horizontal, isEditorLayout ? 14 : 12) - .padding(.vertical, isEditorLayout ? 10 : 8) - .background(isEditorLayout ? AnyShapeStyle(AgentPanelStyle.context) : AnyShapeStyle(LitheTheme.accent.opacity(0.18)), - in: RoundedRectangle(cornerRadius: isEditorLayout ? 16 : 10)) + .frame(maxWidth: .infinity, alignment: .leading) + } else { + userBubble } - .onHover { isHovered = $0 } case .agent: VStack(alignment: .leading, spacing: 2) { AgentMarkdownMessage(text: message.text) @@ -575,4 +607,203 @@ private struct AgentMessageRow: View { AgentToolGroupView(messages: [message], searchText: "", onOpenFile: onOpenFile) } } + + /// Attached image files and replayed pictures show as thumbnails that open + /// the viewer; the text keeps the user's words and non-image attachments. + private var userBubble: some View { + var content = AgentUserMessageContent.parse("") + if case .bubble(let parsed) = AgentUserMessagePresentation.of(message) { content = parsed } + let images = (message.images + content.images).uniquedByID + return HStack(alignment: .bottom, spacing: 4) { + Spacer(minLength: 40) + if !content.text.isEmpty { + AgentCopyButton(text: content.text, help: "Copy message") + .opacity(isHovered ? 1 : 0) + .allowsHitTesting(isHovered) + } + VStack(alignment: .trailing, spacing: 6) { + if !images.isEmpty { AgentUserImages(images: images, hiddenCount: content.hiddenImageCount) } + if !content.text.isEmpty { Text(content.text) } + if let delivery = message.delivery { + Group { + switch delivery { + case .sending: Text("Sending…") + case .received: Text("Agent received") + case .failed(let reason): Text(AgentLocalization.message(reason, locale: locale)).foregroundStyle(LitheTheme.error) + } + } + .font(LitheTheme.uiFont(size: 10)) + .foregroundStyle(AgentPanelStyle.secondary) + } + } + .font(LitheTheme.uiFont(size: isEditorLayout ? 14 : 13)) + .textSelection(.enabled) + .padding(.horizontal, isEditorLayout ? 14 : 12) + .padding(.vertical, isEditorLayout ? 10 : 8) + .background(isEditorLayout ? AnyShapeStyle(AgentPanelStyle.context) : AnyShapeStyle(LitheTheme.accent.opacity(0.18)), + in: RoundedRectangle(cornerRadius: isEditorLayout ? 16 : 10)) + } + .onHover { isHovered = $0 } + } +} + +/// How a user message is drawn, parsed once. The regex scans touch the whole text, +/// and rows re-evaluate on hover and while the transcript streams, so the result is cached. +private enum AgentUserMessagePresentation { + case markup([AgentClaudeCodeEntry]) + case bubble(AgentUserMessageContent) + + private final class Box { let value: AgentUserMessagePresentation; init(_ value: AgentUserMessagePresentation) { self.value = value } } + private static let cache: NSCache = { + let cache = NSCache() + cache.countLimit = 500 + return cache + }() + + static func of(_ message: AgentConversationMessage) -> Self { + let key = "\(message.id)#\(message.text.utf8.count)" as NSString + if let cached = cache.object(forKey: key) { return cached.value } + let value: Self + if let entries = AgentClaudeCodeMarkup.parse(message.text) { + value = .markup(entries) + } else { + value = .bubble(AgentUserMessageContent.parse(AgentClaudeCodeMarkup.removingModelOnlyBlocks(message.text))) + } + cache.setObject(Box(value), forKey: key) + return value + } +} + +extension AgentTaskNotification { + private static let shellSummary = try! NSRegularExpression( + pattern: "^Background command \"(.+)\" (?:completed|failed) \\(exit code (-?\\d+)\\)$") + + /// One line for the transcript. Claude Code words shell-task summaries in + /// English (`Background command "…" completed (exit code 0)`); those are + /// shown in the interface language, and any other summary is kept verbatim. + func displayText(locale: Locale) -> String { + guard let summary else { + return AgentLocalization.string(isFailure ? "Background task ended" : "Background task finished", locale: locale) + } + guard let match = Self.shellSummary.firstMatch(in: summary, range: NSRange(summary.startIndex..., in: summary)), + let name = Range(match.range(at: 1), in: summary), let code = Range(match.range(at: 2), in: summary) else { + return summary + } + if summary[code] == "0" { + return String(format: AgentLocalization.string("Background command finished: %@", locale: locale), String(summary[name])) + } + return String(format: AgentLocalization.string("Background command ended (exit code %@): %@", locale: locale), + String(summary[code]), String(summary[name])) + } +} + +/// Claude Code's own markup in a user message, shown as a quiet status line or output block. +struct AgentClaudeCodeEntryRow: View { + @Environment(\.locale) private var locale + let entry: AgentClaudeCodeEntry + + var body: some View { + switch entry { + case .taskNotification(let notification): + let failed = notification.isFailure + line(systemImage: failed ? "exclamationmark.circle" : "checkmark.circle", isError: failed, + text: notification.displayText(locale: locale)) + .help(notification.outputFile ?? "") + .accessibilityIdentifier("agent-task-notification") + case .command(let name, let arguments): + line(systemImage: "chevron.right.square", isError: false, + text: [name, arguments].compactMap { $0 }.joined(separator: " "), monospaced: true) + .accessibilityIdentifier("agent-command-entry") + case .shell(let command): + line(systemImage: "terminal", isError: false, text: command, monospaced: true) + .accessibilityIdentifier("agent-shell-entry") + case .output(let text, let isError): + Text(verbatim: text) + .font(LitheTheme.uiFont(size: 12, design: .monospaced)) + .foregroundStyle(isError ? LitheTheme.error : LitheTheme.secondaryText) + .lineLimit(12) + .textSelection(.enabled) + .padding(.horizontal, 10) + .padding(.vertical, 6) + .frame(maxWidth: .infinity, alignment: .leading) + .background(LitheTheme.raised.opacity(0.5), in: RoundedRectangle(cornerRadius: 8)) + .accessibilityIdentifier("agent-command-output") + case .interrupted(let text): + line(systemImage: "pause.circle", isError: false, text: text) + .accessibilityIdentifier("agent-interrupted-entry") + case .summary(let text, let tools): + AgentCollapsedBlock(title: AgentLocalization.string("Conversation continued from a summary", locale: locale), + systemImage: "text.append", detail: tools, text: text) + .accessibilityIdentifier("agent-summary-entry") + case .skill(let name, let body): + AgentCollapsedBlock(title: String(format: AgentLocalization.string("Skill loaded: %@", locale: locale), name), + systemImage: "sparkles", detail: nil, text: body) + .accessibilityIdentifier("agent-skill-entry") + case .context(let name, let body): + AgentCollapsedBlock(title: String(format: AgentLocalization.string("Context: %@", locale: locale), name), + systemImage: "tray.full", detail: nil, text: body) + .accessibilityIdentifier("agent-context-entry") + } + } + + private func line(systemImage: String, isError: Bool, text: String, monospaced: Bool = false) -> some View { + HStack(spacing: 6) { + Image(systemName: systemImage) + .font(LitheTheme.uiFont(size: 11)) + .foregroundStyle(isError ? LitheTheme.error : LitheTheme.tertiaryText) + Text(verbatim: text) + .font(monospaced ? LitheTheme.uiFont(size: 12, design: .monospaced) : LitheTheme.uiFont(size: 12.5)) + .lineLimit(2) + } + .foregroundStyle(LitheTheme.secondaryText) + .textSelection(.enabled) + } +} + +/// Long Claude Code-owned text (recap, skill instructions, host context) behind a one-line header. +private struct AgentCollapsedBlock: View { + let title: String + let systemImage: String + let detail: String? + let text: String + @State private var expanded = false + + var body: some View { + VStack(alignment: .leading, spacing: 6) { + Button { expanded.toggle() } label: { + HStack(spacing: 6) { + Image(systemName: systemImage) + .font(LitheTheme.uiFont(size: 11)) + .foregroundStyle(LitheTheme.tertiaryText) + Text(verbatim: title).lineLimit(1) + Image(systemName: "chevron.down") + .font(LitheTheme.uiFont(size: 9, weight: .semibold)) + .rotationEffect(.degrees(expanded ? 0 : -90)) + } + .font(LitheTheme.uiFont(size: 12.5)) + .foregroundStyle(LitheTheme.secondaryText) + .contentShape(Rectangle()) + } + .buttonStyle(.litheNoPress) + .lithePointer() + .accessibilityValue(expanded ? Text("Expanded") : Text("Collapsed")) + if expanded { + VStack(alignment: .leading, spacing: 6) { + if let detail { + Text(verbatim: detail) + .font(LitheTheme.uiFont(size: 11, design: .monospaced)) + .foregroundStyle(LitheTheme.tertiaryText) + } + AgentMarkdownMessage(text: text, size: 12) + .equatable() + .foregroundStyle(LitheTheme.secondaryText) + } + .padding(.horizontal, 10) + .padding(.vertical, 8) + .frame(maxWidth: .infinity, alignment: .leading) + .background(LitheTheme.raised.opacity(0.5), in: RoundedRectangle(cornerRadius: 8)) + .textSelection(.enabled) + } + } + } } diff --git a/macos/Sources/LitheAgentConversationModule/Application/AgentConnectionModel.swift b/macos/Sources/LitheAgentConversationModule/Application/AgentConnectionModel.swift index da6acfe58..583b984e5 100644 --- a/macos/Sources/LitheAgentConversationModule/Application/AgentConnectionModel.swift +++ b/macos/Sources/LitheAgentConversationModule/Application/AgentConnectionModel.swift @@ -124,6 +124,9 @@ public final class AgentConnectionModel: ObservableObject { /// square of the history. The replay lands in `conversations` once, when it ends. private var replayMessages: [String: [AgentConversationMessage]] = [:] private var replayToolIndex: [String: [String: Int]] = [:] + /// Upstream id of the user message a replay last appended to; consecutive user + /// messages are separate entries even though ACP delivers them as chunks. + private var replayUserMessageID: [String: String] = [:] /// Locally prepared sessions without a submitted prompt or upstream history evidence. /// Codex does not persist their rollout until the first prompt, so they cannot be resumed. private var unpromptedSessionIDs: Set = [] @@ -964,6 +967,15 @@ public final class AgentConnectionModel: ObservableObject { case "plan": guard let plan = AgentPlan.parse(update) else { return } conversations[sessionID, default: AgentConversation()].plan = plan.entries.isEmpty ? nil : plan + // Live plans are pinned above the composer. A replay instead keeps every + // snapshot in order, so a history whose Agent wrote little prose (Claude + // Desktop reports progress this way) still shows what each turn did. + if conversations[sessionID]?.isLoading == true, subagentSessionID == nil, !plan.entries.isEmpty { + flushPendingText() + var message = AgentConversationMessage(role: .agent, text: "") + message.plan = plan + replayMessages[sessionID, default: []].append(message) + } // A valid plan proves recovery without implying streamed reasoning. if conversations[sessionID]?.responsePhase == .retrying { markResponseProgress(.waiting, in: sessionID) @@ -990,9 +1002,11 @@ public final class AgentConnectionModel: ObservableObject { // Live input is already shown locally, including pending steering. // Replay still rebuilds the upstream-owned transcript while loading. guard conversations[sessionID]?.isResponding != true else { return } - guard let text = Self.text(of: update) else { return } + let image = (update["content"] as? [String: Any]).flatMap(AgentImage.parse(block:)) + guard let text = Self.text(of: update) ?? (image == nil ? nil : "") else { return } flushPendingText() - append(text, role: .user, to: sessionID, subagentSessionID: subagentSessionID) + append(text, role: .user, to: sessionID, subagentSessionID: subagentSessionID, image: image, + messageID: update["messageId"] as? String) case "tool_call", "tool_call_update": guard let toolCallID = update["toolCallId"] as? String else { return } flushPendingText() @@ -1211,22 +1225,38 @@ public final class AgentConnectionModel: ObservableObject { private func commitReplay(_ sessionID: String) { guard let messages = replayMessages.removeValue(forKey: sessionID) else { return } replayToolIndex[sessionID] = nil + replayUserMessageID[sessionID] = nil conversations[sessionID, default: AgentConversation()].messages = messages } private func discardReplay(_ sessionID: String) { replayMessages[sessionID] = nil replayToolIndex[sessionID] = nil + replayUserMessageID[sessionID] = nil } - private func append(_ text: String, role: AgentConversationMessage.Role, to sessionID: String, subagentSessionID: String? = nil) { + private func append(_ text: String, role: AgentConversationMessage.Role, to sessionID: String, + subagentSessionID: String? = nil, image: AgentImage? = nil, messageID: String? = nil) { + func attach(_ image: AgentImage?, to message: inout AgentConversationMessage) { + if let image, message.images.count < AgentImage.maximumPerMessage, + !message.images.contains(where: { $0.id == image.id }) { message.images.append(image) } + } if conversations[sessionID]?.isLoading == true { - if let last = replayMessages[sessionID]?.indices.last, replayMessages[sessionID]?[last].role == role, + // Two user messages in a row (a prompt after a background notice, say) must + // not fuse into one bubble, which would also stop the notice being recognised. + var startsNewMessage = false + if role == .user, let messageID { + startsNewMessage = replayUserMessageID[sessionID].map { $0 != messageID } ?? false + replayUserMessageID[sessionID] = messageID + } + if !startsNewMessage, let last = replayMessages[sessionID]?.indices.last, replayMessages[sessionID]?[last].role == role, replayMessages[sessionID]?[last].subagentSessionID == subagentSessionID { replayMessages[sessionID]?[last].text += text + attach(image, to: &replayMessages[sessionID]![last]) } else { var message = AgentConversationMessage(role: role, text: text) message.subagentSessionID = subagentSessionID + attach(image, to: &message) replayMessages[sessionID, default: []].append(message) } return @@ -1235,9 +1265,12 @@ public final class AgentConnectionModel: ObservableObject { if let last = conversation.messages.indices.last, conversation.messages[last].role == role, conversation.messages[last].subagentSessionID == subagentSessionID { conversation.messages[last].text += text + attach(image, to: &conversation.messages[last]) } else { - conversation.messages.append(AgentConversationMessage(role: role, text: text)) - conversation.messages[conversation.messages.count - 1].subagentSessionID = subagentSessionID + var message = AgentConversationMessage(role: role, text: text) + message.subagentSessionID = subagentSessionID + attach(image, to: &message) + conversation.messages.append(message) } conversations[sessionID] = conversation } diff --git a/macos/Sources/LitheAgentConversationModule/Application/AgentConversationState.swift b/macos/Sources/LitheAgentConversationModule/Application/AgentConversationState.swift index fd745f835..9424b1a53 100644 --- a/macos/Sources/LitheAgentConversationModule/Application/AgentConversationState.swift +++ b/macos/Sources/LitheAgentConversationModule/Application/AgentConversationState.swift @@ -33,8 +33,12 @@ public struct AgentConversationMessage: Identifiable, Equatable, Sendable { public var delivery: Delivery? public var toolStatus: ToolStatus? public var toolDetails = AgentToolDetails() + /// Pictures carried by a replayed user message; live attachments are parsed from its text. + public var images: [AgentImage] = [] /// Child output is retained with its owner, separate from the parent's reply. public var subagentSessionID: String? + /// Plan snapshot recorded in a replayed history; such an Agent entry has no text. + public var plan: AgentPlan? public init(id: String = UUID().uuidString, role: Role, text: String, toolStatus: ToolStatus? = nil) { self.id = id diff --git a/macos/Sources/LitheAgentConversationModule/Application/AgentHistoryFeatureModel.swift b/macos/Sources/LitheAgentConversationModule/Application/AgentHistoryFeatureModel.swift index 3d3717238..f36060bc7 100644 --- a/macos/Sources/LitheAgentConversationModule/Application/AgentHistoryFeatureModel.swift +++ b/macos/Sources/LitheAgentConversationModule/Application/AgentHistoryFeatureModel.swift @@ -158,9 +158,14 @@ public struct AgentHistoryDocument: Sendable { case .tool: role = String(localized: "Tool") } // Reasoning is exported as a quote so it stays distinct from the reply. - let body = message.role == .thought - ? message.text.components(separatedBy: .newlines).map { "> " + $0 }.joined(separator: "\n") - : message.text + // A replayed plan snapshot is exported as a task list. + let body = if let plan = message.plan { + plan.entries.map { "- [\($0.status == .completed ? "x" : " ")] \($0.content)" }.joined(separator: "\n") + } else if message.role == .thought { + message.text.components(separatedBy: .newlines).map { "> " + $0 }.joined(separator: "\n") + } else { + message.text + } var content = "## \(role)\n\n\(body)" if let status = message.toolStatus { content += "\n\nStatus: \(status.rawValue)" } for detail in [message.toolDetails.input, message.toolDetails.output].compactMap({ $0 }) { diff --git a/macos/Sources/LitheAgentConversationModule/Application/AgentImage.swift b/macos/Sources/LitheAgentConversationModule/Application/AgentImage.swift new file mode 100644 index 000000000..e0c915981 --- /dev/null +++ b/macos/Sources/LitheAgentConversationModule/Application/AgentImage.swift @@ -0,0 +1,351 @@ +import CryptoKit +import Foundation +import ImageIO + +/// An image the Agent produced or the user attached, shown beside the tool call +/// or message that carries it. Inline bytes stay bounded; file references are +/// decoded lazily by the view layer so replaying a long history never decodes pixels. +public struct AgentImage: Identifiable, Equatable, Sendable { + public enum Source: Sendable { + case data(Data) + /// Absolute path of an image on disk, such as a screenshot the Agent read. + case file(String) + } + + public let id: String + public var title: String? + public let source: Source + + /// Larger payloads are dropped: they would sit in the transcript state for the whole session. + public static let maximumBytes = 10_000_000 + /// One tool result rarely returns more; the cap keeps a runaway tool from filling memory. + public static let maximumPerMessage = 24 + public static let fileExtensions: Set = ["png", "jpg", "jpeg", "gif", "heic", "tif", "tiff", "webp", "bmp"] + + public init(data: Data, title: String? = nil) { + self.id = "data:\(data.count):\(Self.fingerprint(data))" + self.title = title + self.source = .data(data) + } + + public init(filePath: String, title: String? = nil) { + self.id = "file:\(filePath)" + self.title = title ?? (filePath as NSString).lastPathComponent + self.source = .file(filePath) + } + + // Identity is the content fingerprint or path; comparing megabytes of pixels + // on every transcript diff would make scrolling pay for each image. + public static func == (lhs: Self, rhs: Self) -> Bool { lhs.id == rhs.id && lhs.title == rhs.title } + + public static func isImageFile(_ path: String) -> Bool { + fileExtensions.contains((path as NSString).pathExtension.lowercased()) + } + + /// Reads an ACP `image` content block: base64 `data` with `mimeType`, or a local `file:` `uri`. + static func parse(block: [String: Any]) -> Self? { + guard block["type"] as? String == "image" else { return nil } + let uri = block["uri"] as? String + let uriTitle = uri.flatMap { URL(string: $0)?.lastPathComponent }.flatMap { $0.isEmpty ? nil : $0 } + if let encoded = block["data"] as? String, !encoded.isEmpty { + let mime = (block["mimeType"] as? String)?.lowercased() + if let mime, !mime.hasPrefix("image/") { return nil } + // Base64 inflates by a third; refuse before allocating the decoded copy. + guard encoded.utf8.count <= maximumBytes / 3 * 4, + let data = Data(base64Encoded: encoded, options: .ignoreUnknownCharacters), !data.isEmpty, + data.count <= maximumBytes, isDecodable(data) else { return nil } + return Self(data: data, title: uriTitle) + } + if let uri, let url = URL(string: uri), url.isFileURL, isImageFile(url.path) { + return Self(filePath: url.path) + } + return nil + } + + /// Reads only the container header, so a payload that no decoder recognises + /// never becomes a thumbnail that stays blank. + private static func isDecodable(_ data: Data) -> Bool { + guard let source = CGImageSourceCreateWithData(data as CFData, nil) else { return false } + return CGImageSourceGetCount(source) > 0 + } + + /// SHA-256 is hardware accelerated, so even a 10 MB payload hashes in a few + /// milliseconds on the main actor; a byte loop over `Data` is far slower. + private static func fingerprint(_ data: Data) -> String { + SHA256.hash(data: data).prefix(12).map { String(format: "%02x", $0) }.joined() + } +} + +extension Sequence where Element == AgentImage { + /// Keeps the first of each id. `ForEach` and the viewer's paging need unique ids, + /// and the same screenshot read twice would otherwise appear twice. + public var uniquedByID: [AgentImage] { + var seen = Set() + return filter { seen.insert($0.id).inserted } + } +} + +/// A user message's text split from the image attachments it lists. +public struct AgentUserMessageContent: Equatable, Sendable { + public var text: String + public var images: [AgentImage] + /// Image references beyond `AgentImage.maximumPerMessage`: removed from the text, not shown. + public var hiddenImageCount = 0 + + private static let attachmentMarker = "📎 " + + /// `[@name](file:///path.png)` as Claude Code replays a referenced file, or a bare `file://` URL. + private static let fileLink = try! NSRegularExpression( + pattern: #"(?:\[[^\]\n]*\]\((file://[^)\n]+)\)|(file://[^\s)\]]+))"#) + + /// Claude Code's image placeholders: `[Image: source: /path]` names the file, + /// `[Image: original 3840x2486, displayed at ...]` only helps the model map coordinates. + private static let imagePlaceholder = try! NSRegularExpression( + pattern: #"\[Image: (?:source: ([^\]\n]+)|original [^\]\n]*)\]"#) + + /// The composer's `📎 /path` line, with its line break. + private static let attachmentLine = try! NSRegularExpression(pattern: "^📎 ([^\\n]*)(?:\\n|$)", options: .anchorsMatchLines) + + /// One image reference found in the message: where it sits and the file it names. + /// `path` is nil for a placeholder that carries no file; it is removed without a thumbnail. + private struct Reference { + var range: NSRange + var path: String? + } + + /// Image attachments become thumbnails; other attachments and the user's own + /// words stay as text. Two shapes occur: the composer's `📎 /path` lines, and + /// file links (`[@name](file:///path)`, bare `file://` URLs) that an agent + /// replays for a referenced file, which would otherwise show as raw addresses. + public static func parse(_ message: String) -> Self { + guard message.contains(attachmentMarker) || message.contains("file://") || message.contains("[Image: ") else { + return Self(text: message, images: []) + } + let ns = message as NSString + let whole = NSRange(location: 0, length: ns.length) + func isImagePath(_ path: String) -> Bool { path.hasPrefix("/") && AgentImage.isImageFile(path) } + func path(ofFileURL raw: String) -> String? { + let trimmed = raw.trimmingCharacters(in: .whitespaces) + if let url = URL(string: trimmed), url.isFileURL { return url.path } + // An unencoded space makes the address an invalid URL; decode it by hand. + return trimmed.hasPrefix("file://") ? String(trimmed.dropFirst("file://".count)).removingPercentEncoding : nil + } + + // Every candidate is located in the original text, so thumbnails follow reading order. + var found: [Reference] = [] + for match in fileLink.matches(in: message, range: whole) { + let raw = ns.substring(with: match.range(at: match.range(at: 1).location != NSNotFound ? 1 : 2)) + if let filePath = path(ofFileURL: raw), isImagePath(filePath) { found.append(Reference(range: match.range, path: filePath)) } + } + for match in imagePlaceholder.matches(in: message, range: whole) { + let source = match.range(at: 1) + if source.location == NSNotFound { + found.append(Reference(range: match.range, path: nil)) + } else if isImagePath(ns.substring(with: source)) { + found.append(Reference(range: match.range, path: ns.substring(with: source))) + } + } + for match in attachmentLine.matches(in: message, range: whole) { + let value = ns.substring(with: match.range(at: 1)) + if isImagePath(value) { found.append(Reference(range: match.range, path: value)) } + } + found.sort { $0.range.location < $1.range.location } + + var images: [AgentImage] = [] + var seen = Set() + var hidden = 0 + var removed: [NSRange] = [] + for reference in found { + // An earlier reference already claimed this text (a link inside a 📎 line). + if let last = removed.last, NSMaxRange(last) > reference.range.location { continue } + if let path = reference.path { + if seen.insert((path as NSString).standardizingPath).inserted { + if images.count < AgentImage.maximumPerMessage { images.append(AgentImage(filePath: path)) } else { hidden += 1 } + } + } + removed.append(reference.range) + } + guard !removed.isEmpty else { return Self(text: message, images: []) } + var working = message + for range in removed.reversed() { + if let swift = Range(range, in: working) { working.replaceSubrange(swift, with: "") } + } + guard !images.isEmpty || hidden > 0 else { return Self(text: message, images: []) } + let text = working.replacingOccurrences(of: "[ \\t]+\\n", with: "\n", options: .regularExpression) + .replacingOccurrences(of: "\n{3,}", with: "\n\n", options: .regularExpression) + .trimmingCharacters(in: .whitespacesAndNewlines) + return Self(text: text, images: images, hiddenImageCount: hidden) + } +} + +/// Claude Code reports finished background work as a `` block +/// inside a user message. It is protocol markup, not something the user typed. +public struct AgentTaskNotification: Equatable, Sendable { + public var taskID: String? + public var status: String? + public var summary: String? + public var outputFile: String? + + /// Statuses that mean the task did not finish normally. + public var isFailure: Bool { + ["failed", "error", "killed", "stopped"].contains(status?.lowercased() ?? "") + } + + /// Nil unless the whole message consists of one or more notification blocks. + public static func parse(_ message: String) -> [Self]? { + let open = "", close = "" + var rest = message.trimmingCharacters(in: .whitespacesAndNewlines) + guard rest.hasPrefix(open) else { return nil } + var result: [Self] = [] + while !rest.isEmpty { + guard rest.hasPrefix(open), let end = rest.range(of: close) else { return nil } + let body = String(rest[rest.index(rest.startIndex, offsetBy: open.count).. String? { + guard let start = body.range(of: "<\(name)>"), + let end = body.range(of: "", range: start.upperBound..