diff --git a/README.md b/README.md index 6361d10..06a55e0 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,7 @@ The MiniMax Code Agent communicates through the host MCP client. Business reques | [仓库透视 / Repo X-Ray](plugins/yujiachen-y/repo-xray/) | Understand a codebase from its Git history: a code map you can drill into, hotspots and bug magnets, high-risk files, owners and knowledge loss, commit rhythm and firefighting, and the tech stack. Works with GitHub, GitLab, and local repositories; analysis runs on your machine | [yujiachen-y](https://github.com/yujiachen-y) | | [Disk Insight](plugins/zch0227/disk-insight/) | See where disk space went on macOS and Windows with a sunburst space map, find junk files sorted into three safety tiers, large and duplicate files, and how much code repositories and Git worktrees take; ask an Agent about any item, and permanently delete what you select after a confirmation that lists every item. Requires MiniMax Code 3.1.2 or later | [zch0227](https://github.com/zch0227) | | [A/时差保安 / Timezone Guard](plugins/liuchen19960902/timezone-guard/) | Check the other side's local time before you message: the target city's clock turns into a green "now is fine", yellow "keep it short", or red "go to sleep" verdict. Requires MiniMax Code 3.1.2 or later | [FineCat](https://github.com/LiuChen19960902) | +| [AI Node Canvas](plugins/antianqi/ai-node-canvas/) | An infinite-canvas node graph for MiniMax Code: wire text, image, video and audio runs into a graph, let each result flow downstream, and run media tools on any result | [antianqi](https://github.com/antianqi) |
Preview: Token Usage Board diff --git a/README.zh-CN.md b/README.zh-CN.md index 58b60d6..672ae6c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -77,6 +77,7 @@ MiniMax Code Agent 通过宿主 MCP 客户端与 MiniApp 协作。业务请求 | [仓库透视](plugins/yujiachen-y/repo-xray/README.zh-CN.md) | 从 Git 历史看懂一个代码仓库:可逐层下钻的代码地图、热点与 Bug 磁铁、高风险文件、负责人与知识流失、提交节奏与救火记录、技术栈;支持 GitHub、GitLab 和本机仓库,全部在本机分析 | [yujiachen-y](https://github.com/yujiachen-y) | | [磁盘透视](plugins/zch0227/disk-insight/README.zh-CN.md) | 查看 macOS 和 Windows 的磁盘空间去向:旭日图空间地图,按三档风险分类的垃圾文件,大文件和重复文件,代码仓库和 Git worktree 的占用;任何条目都可以问 Agent,选中的条目在确认框里逐一列出,确认后直接永久删除。需要 MiniMax Code 3.1.2 或更高版本 | [zch0227](https://github.com/zch0227) | | [A/时差保安](plugins/liuchen19960902/timezone-guard/README.zh-CN.md) | 发消息前先看对方那边几点:按对方城市当地时间给出「现在可以聊 / 短聊就好 / 先睡,别发」三档判定。需要 MiniMax Code 3.1.2 或更高版本 | [FineCat](https://github.com/LiuChen19960902) | +| [AI 节点画布](plugins/antianqi/ai-node-canvas/README.zh-CN.md) | MiniMax Code 里的无限画布节点图:把文本、图片、视频、音频的生成接成一张图,结果顺着连线往下传,任何一张结果都能直接做二次处理 | [antianqi](https://github.com/antianqi) |
预览:Token 用量看板 diff --git a/plugins/antianqi/ai-node-canvas/.minimax-plugin/plugin.json b/plugins/antianqi/ai-node-canvas/.minimax-plugin/plugin.json new file mode 100644 index 0000000..179fc9e --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/.minimax-plugin/plugin.json @@ -0,0 +1,21 @@ +{ + "schemaVersion": 1, + "name": "ai-node-canvas", + "displayName": "节点画布", + "version": "0.82.0", + "description": "无限画布节点编排:把 MiniMax 模型生成的文本、图片、视频串成工作流,连线即数据流,上游产物可直接被下游引用。", + "author": "antianqi", + "icon": "icon.png", + "darkIcon": "icon-dark.png", + "category": "Productivity", + "exampleQueries": [ + "打开节点画布", + "用画布生成一张产品海报", + "把这两个节点连起来", + "用提示词增强写一条 H3 视频提示词", + "把画布上的产品图做成九宫格" + ], + "apps": [], + "mcpServers": ["servers.mcp.json"], + "skills": ["skills/ai-node-canvas/SKILL.md"] +} diff --git a/plugins/antianqi/ai-node-canvas/LICENSE b/plugins/antianqi/ai-node-canvas/LICENSE new file mode 100644 index 0000000..4bea20a --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 antianqi + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/plugins/antianqi/ai-node-canvas/README.md b/plugins/antianqi/ai-node-canvas/README.md new file mode 100644 index 0000000..5ba5830 --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/README.md @@ -0,0 +1,738 @@ +# AI Node Canvas + +[简体中文](README.zh-CN.md) + +An infinite-canvas node graph for MiniMax Code, modelled on the canvas of +`work.batvid.com`. + +**A run lands on a result node.** An image node makes an image, an audio node +makes audio, and pressing 生成 does not fill the card it was pressed on: the +run drops a **结果** node beside it and draws the line — and that result node is +what the next step reads. + +**Two families of node, four kinds of data.** An **input** node holds what goes +in — text, an image, a video or an audio, and the kind is chosen on the card. +A **generating** node produces one of them. A **result** node shows what a +generation produced, and a **group** is a frame around other cards. The kinds +that travel on a line are exactly `text`, `image`, `video` and `audio`: a style +preset and an expanded prompt are text like any other, so neither is a kind of +its own — the style is a text input that was filled from a category template. + +Models are **not** called over HTTP by this package. Every run goes through the +Host Agent, which is already running the user's MiniMax models, so there is no +API key to configure, no credential to store, and no outbound request made by +this package. + +## What it does + +- **Seven kinds of node, and a mode on the card** — a node is a **role** + (输入 / 生成) and the **data it moves** (文本 / 图片 / 视频 / 音频): an + **input** node carries one of the four kinds, and a **generating** node + makes one of the four. A `group` (a frame that carries the nodes inside it) + and a `result` (it appears by itself when a generation finishes) complete + the set of seven. The add menus list the six that can be made — the four + generating nodes, the input and the group — and they are built from the + same table the cards are drawn from, so the menu and the board cannot fall + out of step. A **mode** is another way of making the same kind of data: a + text node writes text, expands a sentence into a prompt (提示词增强), reads a + picture (图生文) or reads a video back into a prompt (视频生文); an audio node + speaks or makes music (文生音乐). The mode is a choice on the card, so it + never gets a row of its own in the menu — and because a mode says what the + node takes, **it also says which input ports the card draws**: 文生图 + waits for a text, 图生图 waits for a picture as well. +- **视频生文 turns a video back into a prompt** — wire a video into a text node + in that mode and it writes the prompt that would produce that video: the + provider's video understanding reads the video shot by shot, and the text + model turns what it saw into one ready-to-use prompt. The text model never + sees a video itself, so the runtime reads the video first and hands the + description over. +- **图生文 does the same for a picture** — the provider's image understanding + reads the picture, and the text model writes the description. Without that + step the text model would only get a url, and it cannot look at one. +- **The text box is the text input** — a generating node has a text box on its + card, and that box is where its text comes in. There is no separate port for + text: a text line lands on the box. When a text line lands there, the box + shows that text and stops being editable, because the text is upstream's now. + The node keeps the prompt you wrote — it comes back when the line is gone — + but it is not sent this time, so what the card shows and what the model + receives are the same thing. +- **Every media node generates itself** — output, prompt editor, parameter row and + a 生成 button live on the same node, so a result can be re-generated, re-tuned + or re-referenced without leaving the node. +- **Category templates** — a text input can be filled from a category template + (商品主图 / S+ 详情 / 社媒封面 / 人像写真) with fields and a live preview; its + text goes downstream like any other text of an input node. +- **Connections are data flow** — the output of every upstream node is injected + into the downstream prompt as clearly marked reference material. Cycles are + refused. +- **Inline prompt editor** — the composer is an inline `contentEditable`, not a + textarea, so `@` references can be embedded in the text as chips with + thumbnails. IME composition is handled explicitly, and editor keystrokes never + reach the canvas shortcuts. +- **Provider-specific reference syntax** — mentions are encoded as + `[[cw:kind:nodeId]]` and compiled per provider family (kling / minimax / + wanxiang / bytedance) into that provider's own syntax, with the asset URLs + collected into separate arrays. +- **Model-aware parameters** — the parameter surface follows the generation mode + *and* the selected model: ratio and resolution for images, plus duration and + "continue from previous shot" for video, plus voice and emotion for audio. + Values a model does not offer are snapped onto ones it does. The ratio menu + says which way the frame lies (16:9 横屏, 9:16 竖屏, 1:1 方形), while the + value the provider takes stays the ratio itself. +- **A global task table** — several nodes may generate at the same time while a + single node refuses to run twice. Every run carries a progress log, and + provider failures are translated into something actionable (402 额度用完 / + 429 限流 / 超时). +- **Human in the loop** — when the Agent asks for a tool permission, the request + is rendered as a card in the run panel. The page can only answer `allowOnce`, + and only after the user picks it. The server downgrades any other decision to + `deny`. +- **Cancel one run at a time** — every run in the progress panel carries its own + Cancel button. Cancelling stops that run only: the other runs keep going, the + Agent call behind it is torn down through its own abort controller, and the + node is free to run again right away. +- **Open in chat** — a run that went through an Agent turn can be opened in the + Host chat side panel, so the conversation can be continued where the user + reads it. The generating node keeps a Chat button for the same session, so a + finished run is still reachable, and a New Chat button that drops the + session so the next run starts a fresh one. +- **Capability panel** — one `/api/palette` call serves the whole surface + (node types, sizes, per-mode parameters, per-model limits, templates, + connector tools) and the UI renders from it, so the panel can never drift from + the server's own rules. The connector tools are the provider's whole + inventory, and every row says whether the Host granted it and whether this + app runs through it — the provider offers more than the app uses, and the + panel does not pretend otherwise. +- **Local media tools** — 14 transformations that run on the machine: 抽帧 / 宫格抽帧 / + 片段剪辑 / 视频剪裁 / 自动分段 / 音画分离 / 合并视频, plus 字幕擦除 / 视频去水印 / + 背景移除, and 智能抠图 / 变清晰 / 无损改尺寸 / 证件照. + They are local transforms, not provider features, so they need `ffmpeg` on + the machine; the panel says so when it is missing. + Four of the reference product's image tools are deliberately left out — 拼图 + (it needs several pictures, not one), 海报 (the layout comes out too rough), + 去水印 (the user has to place a rectangle first) and 服装换色 (it shifts the + hue of the whole picture, background included). A tool on a result node is a + shortcut for that one picture, so it has to work on that picture alone. +- **Video understanding** — 视频拆解 / 爆款分析 read the video through the + provider's own understanding tool and leave a text node behind. +- **Three roles, three kinds of node** — an **input** node holds what goes in + (text and local files), a **generating** node holds the prompt, the model and + the parameters, and a **result** node appears next to it when the run + finishes. A generating node has no result of its own, so the chain reads + left to right: input → generate → result → generate → result. +- **Local files** — an input node takes an image, a video or an audio file + from the machine. The file is kept under the runtime's own data directory + and served from there; when it is used as a generation reference it is + uploaded to the provider first, because the provider cannot reach the + loopback address. +- **Drag to empty space** — pulling a line from a node's right port into + empty space opens a picker with the node types that can actually take what + that node offers, and the one you pick is created there and wired up. +- **Ports follow the mode** — a generating node has one port per input kind + it can take, and the kinds follow the model it selected: an image model + offers a text and a picture slot, a video model that takes references + offers text, picture, video and audio. Pulling a line from an input port + into empty space creates the input node that slot needs, already wired up. + A line starts and ends on the ports themselves, so a node with several + ports has one line per port. +- **Every card declares its sockets** — the strip under the header says what + the node takes and what it hands on, the way a ComfyUI node names its + inputs and outputs: what goes in on the left, what comes out on the right. + The kinds follow the model, so a card never offers a socket it could never + fill. **An input that is still waiting is a hollow ring with `待接入` next + to it**, a plugged-in input is a filled dot, and an output is the kind the + node produces — so what is still missing can be read off the card without + touching anything. +- **The mode follows the inputs** — a picture on an image node turns it into a + 图生图 node, and the options that need it only show when it is there. +- **A prompt node** — a sentence and a template, expanded by the text model + into a prompt that meets the template. The template is picked from a + drop-down of presets or written by hand. Besides the shop and social + presets, the drop-down offers one shape per **MiniMax H3 generation mode**: + the three-field format for each of the four base tasks (T2VA / I2VA / + FL2VA / L2VA) and the six-section full-reference format, taken from + MiniMax's own prompt-writing guides. The structure is written in English and + the dialogue keeps its own language. +- **The pictures are named the way the request names them** — an `@` in the + prompt points at a result on the board, and the runtime compiles it to the + provider's own reference syntax (`@图片1`, `@视频1`, `@音频1`). **The number + is the place the material has in the request**, so the picture the prompt + calls `@图片1` is the first picture the request sends — the prompt and the + request can never mean two different pictures. The Agent that writes the + prompt is told the same numbers, so what it writes binds to the same + material. +- **The `@` menu offers the whole board** — not only the chain above the node. + A prompt may be about a picture that sits somewhere else, and naming it is + what brings it in. The ones above the node come first. +- **Naming a material wires it in** — a mention says "this prompt is about + that material", so the node that generates from the prompt has to receive it + as an input of its own. The app does that: an **输入** node is created to + hold the material, and a line carries it to every node that takes that kind + and does not have it yet. The user does not draw that line a second time, + and the same picture is never held twice. The node that names the material + then names the holder — the one the request actually carries — so the + number in the prompt and the picture in the request are the same picture. +- **The alignment line belongs to the runtime** — MiniMax's H3 guide puts one + line on the very top of the prompt of a picture-to-video request, saying + which picture is the first frame and which is the last one. The runtime + writes it, because the runtime is the one that knows what the request + carries and how long the clip is: `For the target video, at 0.00 seconds + into the target video, @图片1 (from [Shot 1]) is fully referenced.` for a + first-frame request, and the two-picture form for a first+last one, with + the length of the clip and the index of its last shot filled in. The + templates do not write it, so the prompt never carries two of them. +- **A picture the request will not send is refused** — a first-frame request + carries one picture and the provider cannot take a reference set at the + same time, so naming another picture in that prompt would leave a number + pointing at something the model never sees. The run says so and stops + instead of sending it. +- **A number that is out of range is refused** — the same rule, checked on + the text that reaches the provider: a prompt that says "照着 @图片2 写" + while the request carries one picture would quietly produce something + else. The run is refused with the reason, so the mismatch is fixed instead + of being paid for. +- **A generated text takes over the prompt** — when a text result is + connected upstream, the node's own prompt box turns into a read-only panel + showing the text it was given, because there is nothing left to type. +- **A result can be handed to the user** — the file is on this machine (the + runtime keeps it so the canvas can point at it), and the card has a **保存** + button that hands it over, named after the node it belongs to. The right + click menu says the same thing. +- **There is a library of the local files** — everything the runtime + produced and everything the user picked sits in a folder on this machine. + The **资产库** panel shows it: an asset can be **插入** back into a flow as + an input node, **保存** to the machine, or **删除** when no canvas uses it + any more. A file that a canvas still points at is never deleted — the + library says who uses it. **清理未引用** takes away the files nothing + points at, so the folder does not grow for ever. +- **The result sits in the middle of the chain** — when a generating node + feeds another generating node, its result is placed between the two and the + line is rewired through it, so the chain reads 生成 → 结果 → 生成 and the + next node reads the result. +- **A chain runs itself** — pressing 生成 on the last node of a chain runs the + generating nodes upstream that still owe a result, deepest first. A node + that already has a result is used as it is and never run again. Running a + step again replaces its result instead of piling up a second one. +- **Several assets in one run** — there is a **数量** (count) drop-down next to the + 生成 button, with 1 / 2 / 3 / 4 / 6 / 8. Asking for four gives four versions. + **The Host stops a connector call that takes too long (30 seconds by default)**, + so four assets are **four calls of one asset each** — every call gets its own + window — and not one request carrying four of them, which would run into the + limit and may already have been charged. **Every asset gets its own card, + laid out in one row**, so the versions can be compared side by side; when one + of them fails the others still show up and the run log says which one is + missing. A video is one task per clip and a text is one Agent turn, so + neither of them is batched and those two modes have no count to choose. +- **One slow call at a time** — the window belongs to a single call, so a call + that shares the machine with another one is the call that runs over it. The + slow calls (pictures, speech, music, video submits) therefore wait for each + other: the second angle of one picture starts when the first angle is done, + and each of them gets a window of its own. A call that fails does not stop + the ones behind it. The run log says how long a call waited, and a failed + call says it too, so "the provider was slow" and "it waited its turn" are + never confused. +- **Cards are spaced by their own size** — the results of one run, the tiles + of a 宫格切分 and the children a tool produces are all placed by one rule: the + distance between two cards is the size of a card plus a gap. The size of a + card is the one the page measures — the preview follows the media's own + ratio and the tool bar wraps — so a batch is placed again once its cards are + measured. Two result cards that are sixty pixels apart cover each other, and + the picture under them is then hidden. +- **A card never covers another one** — a card that lands on another one + pushes it aside, to the right, where there is room. The result of one + 多角度 branch would sit on top of the branch next to it; the branch steps + aside instead. +- **A material this machine has lost is left out** — the chain above a node + is the reference, and a file that is not on this machine any more cannot be + sent to the provider either. The run says which material was left out and + goes on with the rest, instead of dying on it. +- **A palette that follows the system** — the page follows the system theme + until the user picks a side, and the choice is remembered. +- **The Agent can read the canvas** — this Mini App serves a hosted MCP server + (`servers.mcp.json`), so the Agent gets tools of its own: `canvas_list` names + the canvases, `canvas_read` returns the nodes, the connections and the + slots of one — the same three states the card shows — and + `canvas_list_templates` names the ready-made workflows. All three are + read-only. +- **The Agent can change the canvas** — six more tools write to the board: + `canvas_add_node`, `canvas_connect`, `canvas_update_node`, + `canvas_apply_template`, `canvas_remove_node` and `canvas_run`. They change + what the user sees, so they are never read-only and they say so in their + description. +- **The package tells the Agent how to use it** — a Skill + (`skills/ai-node-canvas/SKILL.md`, written in Chinese) says what the canvas + is for, when the **prompt node** is the right step (the H3 templates carry + the provider's own prompt shape, and writing one by hand loses it), how the + reference numbers work, which chains are worth building, and what the + models refuse. The Agent reads it when the request is a generation chain + rather than a single prompt. +- **A template is the skeleton of a chain** — the 模板 menu holds + **seven ready-made workflows**, under the headings they belong to: + - *从素材出发* — a picture **imitated** and a clip **imitated**: the material + is read back into a prompt and the prompt is what is generated from; + - *H3* — one for each shape the request can carry: T2VA (words), I2VA (a + first frame), FL2VA (a first and a last frame), L2VA (a last frame) and the + full-reference set. Each of them carries the **prompt node** that writes the + structure H3 expects — the runtime adds the line that names the picture. + + These seven are **files** (`miniapp/templates/*.json`), not code: a template + is data, the user opens one, changes it and keeps it, and changing one of the + shipped ones is a change to a file rather than to the code. + + A template carries the **steps** only — the nodes, the lines, the prompts and + the parameters — and never what a run left behind, so it works on another + machine and does not come out as a canvas full of broken pictures. Putting one + on the canvas does **not** run it: the money is spent when the user presses + 生成. +- **One button, one menu, two ways in and two ways out** — the toolbar carries a + single **模板** button, and the menu it opens holds: + - **我的模板** — the chains the user kept. They live in the runtime's own + template folder, so they are there whatever happens to a file. The Agent + sees them too: `canvas_list_templates` lists them, `canvas_apply_template` + puts one on the canvas. + - **打开本地模板** — a template file that is on this machine, picked by the + user. + - **保存为我的模板** — the canvas that is open now becomes one of the user's + own, under a name they give it. The steps are kept; the results and the + material are left behind. + - **保存到本地** — the canvas that is open now becomes a file, in any folder + the user names: the Downloads folder, the desktop, the documents, or any + folder the user types. The runtime writes it and says where it went. + + There is no third way in and no second way to keep a file: a folder to look + in, a button that keeps a file, and a second way to pick a file were three + ways to say the same thing. +- **A template that is picked gets a canvas of its own** — the canvas that is + open now is a piece of work in progress and is left alone; a template opens a + new canvas under the name of the template (an empty canvas is used as it + is: there is nothing on it to lose). A file carries the steps and nothing + else, so it opens whole on another machine: no results, no material, nothing + that would break. The empty input a chain starts from **does** travel, so the + person who puts the template on their board puts their own file in it. +- **A template of the user can be managed** — every template of the user + carries a delete button next to it (the question is asked in the page, like + every other question here). The ready-made ones are not the user's to take + away. +- **The canvas is a workbench, and it saves itself** — a change is written + down about a second later, and the state beside the canvas name says + whether it happened, so the toolbar has no **保存** button: a button that + does by hand what the page already does is only a second way to do it. A + window that goes away takes the local draft with it, so a change made + between the timer and the close is still there when the canvas is opened + again. What is worth **keeping** is a **template** (the steps) and the + **results** (in the library, savable to the disk) — and the delete dialog + says so before it takes anything away. +- **The prompt node carries the template** — `canvas_add_node` and + `canvas_update_node` take a `templateId`, so a prompt node the Agent makes + expands the sentence with the same template the page would pick, the H3 + shapes included. The page and the Agent ask one rule for it. +- **The page stays the only writer** — a canvas lives in the page's memory, and + the store refuses a write that is based on an older version of it. So a change + from the Agent is never written to the stored document behind the page's + back. The page **claims** the canvas before it reads it; while the claim is + alive the runtime does not write that canvas, and the change is handed to the + page as an *action*. The page applies it with the very same rule the runtime + would have used — one rule, in one place — and answers, and only then is the + canvas saved. A claim and a write take turns on the same lock, so a change can + never land between the claim and the read. When nobody has the canvas open, + the runtime applies the action itself, and the result of a run it started + lands on the canvas on its own. **Either way the canvas looks the same**, + because the rule that places a node, draws a line and puts down a result is + the same function on both sides. +- **One rule for colour** — the board has two families of colour, and each + means one thing: + - The **content colour** follows the kind of data: amber for text, green + for a picture, blue for video, violet for audio. + The node header, its ports, its socket chips and the line between them + share that colour, so a chain can be read by colour alone. + - The **interaction colour** is always the accent: selection, focus and + hover are it, and it never mixes with a content colour. + - A port carries the colour of what flows through it; an **output is a + filled dot and an input is a hollow ring until something is plugged + into it**, so the empty slots are obvious. + - While a node is selected, **the nodes and lines around it stay bright + and the rest of the board steps back**, so the eye lands on the chain. + - The capability list carries a legend of the data colours. +- **State is never carried by colour alone** — a selected node shows a `Selected` + chip and its accessible name carries `(selected)`; every node is a named + group, the prompt editor has a name of its own, the progress panel is a + `role="status"` area, and a failure banner is a `role="alert"`. +- **The card follows the media** — a generated image or video is previewed at + its own ratio, and the card is sized around that preview instead of + squeezing it into a fixed frame. A tall 9:16 asset gives a narrow, tall + card; a wide 16:9 asset gives a short one. A card the user resized by hand + keeps the width they chose. +- **Undo / redo, multi-select, minimap, draft recovery** — snapshot history over + nodes, connections and viewport; `Shift`-drag marquee; a live overview of the + graph; and a local draft that can be restored after a crash. +- **The page asks in itself** — a new canvas, a new name and a deletion are + settled **inside the board**: the name is typed into the header and the + deletion is confirmed in a dialog that belongs to the page. A native + `prompt()` or `confirm()` is a separate OS window that never reaches the + screen inside the MiniApp host, so a question asked that way is a question + nobody answers — and the button looks as if it did nothing. + +## Connections + +A connection says how material travels, and it also says what a node takes. + +The input ports on the left of a generating node — how many there are and what +kind each one carries — **follow the mode the node is in** (the mode is the +choice on the card: 文生图 / 图生图 …). So: + +- **A line comes in**: if the kind it carries (text / image / video / audio) is + one the current mode does not take, the node **moves to a mode that takes it** + — the mode that takes the new kind *and* everything already connected. A + picture turns 文生图 into 图生图. When no mode takes it, the line is refused + and the reason says which mode would. +- **A mode is picked by hand**: a line the new mode does not take has no port + to land on, so it is removed and the user is told how many went. +- The rule exists once. A line dragged by hand and a line the Agent asks for + through MCP land in exactly the same shape. + +There are four refusals: a node connecting to itself, the same line twice, a +line that would close a cycle, and a line whose kind **no mode of the target +takes**. + +### A picture becomes a video in several ways + +The provider has **several different requests** for turning material into a +video. They are not one request with different parameters: + +| Request | What is sent | Who can run it | +| --- | --- | --- | +| First frame | `input_image` + `reference_type: first_frame` | all three models | +| First+last frame | the first frame + `last_frame_image` | H3 and H3-Max | +| Last frame | `input_image` + `reference_type: last_frame` | H3 and H3-Max | +| Reference set | `reference_images` / `reference_videos` / `reference_audios` | H3 only | + +So the **mode says which request is wanted** and the **model says which +requests exist** — two different questions. The video node has five modes, and +every one of them has an exact list of what it takes: what it takes is what the +request carries. + +| Mode | What it takes | +| --- | --- | +| 文生视频 | a text box | +| 图生视频·首帧 | a text box, a picture (first frame) | +| 图生视频·首尾帧 | a text box, a picture (first frame), a picture (last frame) | +| 图生视频·末帧 | a text box, a picture (last frame) | +| 参考生视频 | a text box, a picture, a clip, a sound | + +- The first+last frame mode needs **two pictures**, so the card has two picture + ports and the slot strip names them **首帧** and **尾帧**; the line that lands + on a port is the picture that request carries. **With only one picture it + falls back to a first-frame request** — the provider does not take a + first+last request that has no last frame. The last-frame mode is the same + request at the other end: the picture is the ending, and it is sent as the + single `input_image` with `reference_type: last_frame`. +- The reference set is **one request**, and the set may hold pictures, videos + and audios together: nine pictures, three videos and three audios, twelve + items in all. So the mode carries all three kinds, and the ports it shows + are the ones the model really takes — H3 shows picture, video and audio, and + a model without a reference set does not show the mode at all. A clip and a + sound are references like a picture is: they tell the model what to follow, + and the soundtrack is still the model's own. +- Three of the provider's own rules are enforced before the run is sent, because + the page cannot see any of them: **a reference set needs at least one item**, + **an audio reference never comes alone** (it needs a picture or a video), and + **the set holds twelve items in all** (nine pictures, three videos and three + audios is fifteen — refused). A run that breaks them is refused with a + reason instead of being sent and coming back with a provider error. +- The ratio control follows the **request**, not the model: the two keyframe + requests follow the frame, so the card offers no ratio there; a + text-to-video run and a reference set can pick one. + +### The options follow the model + +The video node's model decides what the card offers, because the three video +models are not the same model. The numbers come from the connector's +conditional schema, not from taste: + +| | MiniMax-H3 | MiniMax-H3-Max | MiniMax-Hailuo-2.3 | +| --- | --- | --- | --- | +| Modes | 文生视频 · 首帧 · 首尾帧 · 末帧 · 参考 | 文生视频 · 首帧 · 首尾帧 · 末帧 | 文生视频 · 首帧 | +| Duration | 4–15 s | 5–15 s | 6 s or 10 s | +| Resolution | 768P, 2K | 480P, 768P | 768P; 1080P only with 6 s | +| Ratio | 6 ratios + `adaptive` (follows the first frame) | 6 ratios + `adaptive` (it is in the H3 family) | none — it is not in the H3 family | +| References | up to 9 pictures, 3 videos and 3 audios, twelve items in all | one picture, and it is the first frame | one picture, and it is the first frame | +| Sound | renders its own synchronized track | **silent** — use H3 for a voice | silent — add a speech node on top | +| Billing | account credits | account credits | Token Plan | + +So the card follows the model: switching to H3-Max takes the **reference set** +(it has none) and the audio port off the table; switching to Hailuo leaves +only 文生视频 and 首帧. The same rule runs when the Agent changes the model +through MCP, so the canvas is the same whoever asked. + +> H3 and H3-Max are paid from the **account balance**, and the provider +> refuses them while the account has no **按量自动扣费** (pay-as-you-go) turned +> on — so a clip that should have a voice comes back refused instead of +> silent. Hailuo runs on the Token Plan, but it is the silent one. + +> The card has **no sound switches**. There used to be two (sync dubbing, +> ambient sound), but a video run goes straight to the connector and the place +> those switches landed — the Agent's prompt — is not on that path, so +> flipping them did nothing. The sound is decided by two things now: the +> **mode** says whether the request carries an audio reference, and the +> **model** says whether the picture comes with a track (that is in the model +> dropdown's note). + +### The text box is the text input + +A generating node has a text box on its card, and that box is where its text +comes in. The text has **no port of its own**: a text line lands on the box, +not on the edge of the card, and the slot strip does not list the text either — +one thing, one place. + +When a text line lands on the box, the box shows that text and turns read-only: +the text belongs to upstream now, so there is nothing to type there. The node +keeps the prompt you wrote — it comes back when the line is gone — but it is +not sent this time, and the text that handed over is no longer repeated as a +reference. So the card and the run always say the same thing: what you see in +the box is what the model receives. + +What a connection *does* depends on the kinds involved: + +| Upstream | Downstream | Effect | +| --- | --- | --- | +| `text` (an input, a text result, a text node that has run) | a generating node | The text takes over the downstream node's text box: the box shows it and cannot be typed in, and that text is the prompt of this run. | +| `image` / `video` / `audio` | `text` | The output is injected into the Agent turn as a clearly marked reference block. | +| `image` / `video` / `audio` | `image` / `video` / `audio` / `music` | The asset is sent to the provider as a reference, carried by the connection. A mention in the prompt says which part of it the prompt talks about and, for that kind, its own list wins. | +| `group` | any | A group is a frame, not a data source: it carries no output into the downstream node. | + +The provider's per-model limits still apply — nine images, three videos and +three audios at most — and they are checked on the server before a run +starts, so a connection that would exceed them is refused with a clear +message rather than failing at the provider. + +## Install + +Copy the whole `ai-node-canvas` directory, including the hidden `.minimax-plugin` +directory, into your MiniMax Code data directory: + +```text +Repository: ai-node-canvas/ +Install to: /plugins/ai-node-canvas/ +Manifest: /plugins/ai-node-canvas/.minimax-plugin/plugin.json +``` + +The default data directory is `~/.minimax`, so the destination is normally +`~/.minimax/plugins/ai-node-canvas/`. + +Restart MiniMax Code, confirm the plugin is recognised and enabled, then ask the +Agent to open "节点画布" — or open it from the plugin list. + +## Usage + +1. Right-click empty canvas space to add a node, or double-click to drop an + image node. +2. Add an `输入` node, pick `文本`, fill it from a category template and apply + it — the text now applies to everything downstream. +3. Select a node, write a prompt, choose a model and parameters, and press + **生成**. The result replaces the node's own output. +4. Type `@` in a prompt to reference an upstream node's output. +5. Drag the circle on the right edge of a node onto another node to connect + them. + +### Keyboard + +| Keys | Action | +| --- | --- | +| `Ctrl+Z` / `Ctrl+Shift+Z` / `Ctrl+Y` | Undo / redo | +| `Ctrl+C` / `Ctrl+V` | Copy / paste selected nodes | +| `Delete` / `Backspace` | Delete the selection | +| `Shift`+drag | Marquee select | +| `Shift`+click | Toggle one node in the selection | +| `Escape` | Close mention menu → panel → menu → deselect | + +## Tested environment + +- MiniMax Code 3.1.2 on Windows 11 (Node 22) +- MiniMax Code 3.1.2 on macOS 15 +- Requires a desktop client that supports MiniApps and Agent turns. + +## Local media tools + +The 14 media tools are local transformations, so they need `ffmpeg` and +`ffprobe` on the machine. The binary is looked up on `PATH`, in the WinGet +links and package folders, and in the usual install locations; `NXC_FFMPEG` +overrides the search. The binary is not bundled: a packaged ffmpeg is several +times larger than the package size limit. + +When ffmpeg is missing, the tool buttons stay disabled and say what is +missing, and the media status route reports the same. Everything else — the +generating modes, the canvas, the text tools — works without it. + +A card holding a **video** shows no tool bar at all: the finished clip is a +deliverable, not a working material, and a dozen buttons under it are noise. +No empty strip is left under the clip either — no tools, no footer. +The video tools stay in the runtime, so the Agent can still call them through +the MCP server. + +### The tools that draw in the browser + +Two of the tools work in the page rather than on this machine: **宫格切分** +cuts a picture into a grid, and **剪裁** keeps the part of it the user drags +out. They need no ffmpeg and no credit. + +The page draws the result and hands the bytes to the runtime, which publishes +them into the asset folder like any other produced file. The result is +therefore a real file: it can be saved to the user's machine, it shows up in +the asset library, and the next tool can read it. A data URL would have lived +inside the canvas document and nowhere else. **剪裁** leaves a picture of its +own behind, so the original stays where it was. + +### The tools that ask the model + +**多角度** and **打光** run the Agent on the picture. 多角度 takes several +angles, and one run cannot answer two angles with one picture, so the tool +runs once per angle and each run names its own. Every run is a generation, so +it costs the user's credit; the panel says how many runs a selection will take +before the button is pressed. + +### One chain, all the way down + +Every tool leaves a **result node** behind, whichever way it was made. Only a +result node carries the tool bar, so the picture a tool produced can be +worked on again: 智能抠图 → 变清晰 → 宫格切分 is one chain, not three +separate pictures. + +## Data & access + +- **What the connector is for** — the `matrix` connector is used for media + generation only: images, video, speech and music, plus reading a video + back for understanding. The Host holds the credentials and the egress; + this package never touches an account of its own. +- **Files read** — only this package's own page assets, read from `context.pluginRoot` + (`miniapp/client/index.html`, `app.js`, `mention.mjs`, `tools.mjs`, + `forms.mjs`, `actions.mjs`, `styles.css`). + No Host file is read; there is no supported API for that. +- **Files written** — canvas documents only, under `context.dataDir/canvases/*.json`, + written atomically (temporary file in the same directory, then renamed). The media + tools additionally write working files and published results under + `context.dataDir/work/` and `context.dataDir/assets/`, and an uploaded file is + kept under `context.dataDir/uploads/`. +- **The one write outside that folder** — **保存** on a node, or on an asset in the + library, copies the file into the user's **Downloads** folder under the name + the node carries. It happens only when the user asks for it, and the page says + where the file landed. A file that is already there is never overwritten: the + copy gets a number. + **保存到本地** writes there too — the runtime writes the file and + says where it went, because a download the browser starts inside the Host + never arrives under the name it was given. The folder is the one the user + named in the export panel; it has to be a folder that is there, and the file + name has to be one plain name. +- **What is read of a file of the user** — only the file the user picked in + **打开本地模板**, and only when it is a template by the one rule that reads a + template. A file that is something else is refused and never offered. No + folder of the user is scanned: a file is read when the user hands it over. +- **The library reads those two folders** — the asset folder and the upload + folder are what the **资产库** panel lists, and **清理未引用** deletes the + files in them that no canvas points at any more. A file a canvas still + points at is never deleted by the library. +- **Network hosts** — the asset host that the generation provider returns in + `get_asset_url` (the provider's own object storage domain), and the host the + provider's upload command returns a temporary link on. Nothing else. + The Node service downloads a produced asset so the canvas keeps a link that + does not expire, and uploads a local file when it is used as a generation + reference, because the provider cannot reach the loopback address. Every + other request — image, video, audio, music, voice catalogue, video + understanding — goes through the Host connector (`context.hostConnector`), + and the Host owns the credentials and the egress. +- **The two steps of an upload** — the provider's upload command does not take + a local path, because the provider cannot read this machine's disk. It takes + the file name and the media type and answers with a signed form; the runtime + then posts the bytes to the host that form points at, as + `multipart/form-data` (the form's own fields carried over, the file in the + part named by `file_field_name`). The `temp_url` in the answer is the address + the provider can fetch. The form host has to be https (a loopback address + excepted), and that one request is made by the runtime itself rather than + through the Host connector. +- **How long a connector call may take** — the Host puts a timeout on one + connector call (30 seconds by default). A call that runs over it is aborted + and reported as an unknown outcome: it may have been dispatched, and it may + have been charged, so it must never be retried automatically. A slow + generation (an image generation with a reference is the one that hits it + most often) can therefore fail once; the message names the tool and how long + it ran. Video is submitted and then polled, so it is not affected. +- **Processes spawned** — `ffmpeg` and `ffprobe`, and nothing else. Both are + looked up on the machine and run with an argument array, never through a + shell, so no part of a path or a parameter can be read as shell syntax. + The commands are: + - `ffmpeg -hide_banner -loglevel error -y …` — one call per media + tool: a frame extract (`-ss`, `-frames:v 1`), a grid tile + (`fps=…,tile=CxR`), a trim (`-ss`, `-to`, libx264/aac), an aspect crop + (`crop=W:H:X:Y`), a scene-detect pass (`select='gt(scene,…)',showinfo`), + an audio split (`-an` / `-vn`), a concat (`-f concat -safe 0 -i `), + a chroma key (`chromakey=0xRRGGBB`), a logo removal (`delogo=x:y:w:h`), + and the image passes (`colorkey`, `scale`, `pad`, `crop`, `unsharp`, + `hqdn3d`). + - `ffprobe -v error -print_format json -show_streams -show_format ` — + to read the duration, the size, the frame rate and whether the file has + an audio track. + Every parameter that reaches a filter string is validated first: numbers are + coerced and clamped, and every option value comes from a fixed list. A value + from the page can never become filter syntax. The spawned children are + tracked and stopped when the runtime is disposed. +- **Credentials** — none are read, stored, or forwarded. The Host authenticates the + loopback port and strips its credential before a request reaches this package. +- **The hosted MCP server** — `servers.mcp.json` declares one Streamable HTTP + server, mounted on `/mcp/canvas` by `miniapp.json`; the Host rewrites its + address and authenticates it like the rest of the port. The server speaks + stateless JSON-RPC (`initialize`, `ping`, `tools/list`, `tools/call`) and + offers three reading tools and six writing ones: + - `canvas_list` — the canvases with their id, title, node count and last + change. + - `canvas_read` — one canvas: its nodes (type, title, status, model, + parameters, and what went wrong if something did), the connections, and + for every node the slots it declares with the state the card shows + (`filled`, `ready`, `open`, `out`) — a port that has a role says which one + it is, so the first frame and the last frame are two ports. + - `canvas_list_templates` — the ready-made workflows and the templates of + the user's own. + - `canvas_add_node`, `canvas_connect`, `canvas_update_node`, + `canvas_apply_template`, `canvas_remove_node`, `canvas_run` — add a node, + draw a line, change a node, put a workflow on the canvas, remove a node, + run a node once. + + The reading tools only read the canvas documents under + `context.dataDir/canvases/`. They never write, never start a run and never + call the provider, so the Agent can look at the board without touching it. + The writing tools go through the **action** channel: when the page holds the + canvas the change is handed to it and applied there, and when nobody has it + open the runtime applies it itself — **never** a write behind the page's + back. The endpoint needs no dependency of its own: the dependency folder is + not published with the package, so the protocol is spoken by hand. +- **Connector access** — the manifest declares the `matrix` provider under + `hostConnectorAccess`. The Host grants it at install time, and this package + does call it: + - Image, video, audio and music are produced by the Node service itself + through `context.hostConnector` (`generate_image` / + `submit_video_generation` / `query_video_generation` / + `synthesize_speech` / `batch_text_to_music`); the produced asset is turned + into a short-lived display URL with `get_asset_url`, and the drive node id + is stored so an expired link can be refreshed. + - The voice drop-down is live data too: `/api/voices` reads the real TTS + catalogue while handling a request. + - 视频拆解 / 爆款分析 read the video with `videos_understand`. The provider + fetches the asset itself, so those two tools work on a generated video; a + file this MiniApp produced locally is served from the loopback address and + the provider cannot reach it. + - The call path does not change the billing: same connector, same account. + The model decides the pool (H3 and H3-Max use account credits, + Hailuo-2.3 is Token Plan eligible). + - Text has no connector tool, so it keeps the Agent turn. +- **Agent turns** — yes, but only for the text mode and prompt work. Pressing + **生成** in the text mode runs an Agent turn through `context.agent` with the + user's full Agent capabilities: files, shell, Skills, and MCP, including + this Mini App's own tools. The Agent may therefore read and write files, run + shell commands, and call MCP tools on the user's behalf. + Permission requests are surfaced in the run panel and are answered with + `allowOnce` only after the user chooses; the page never answers automatically. The + server downgrades any decision that is not `allowOnce` to `deny`. + Text coming from the page, from files, or from the network is treated as data, not + as instructions. +- **State location** — `context.dataDir`, a private directory the Host creates and + owns for this plugin. The location is treated as opaque and may change between + versions; do not hard-code it. The browser also keeps a local draft in its own + storage for crash recovery. diff --git a/plugins/antianqi/ai-node-canvas/README.zh-CN.md b/plugins/antianqi/ai-node-canvas/README.zh-CN.md new file mode 100644 index 0000000..4142099 --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/README.zh-CN.md @@ -0,0 +1,463 @@ +# AI 节点画布 + +[English](README.md) + +MiniMax Code 里的无限画布节点图。每个节点是一次模型运行,每条连线是一条数据通路—— +把文本节点连到图片节点,上游文本会自动注入下游提示词。 + +模型**不是**由本包直连 HTTP 调用的。图片/视频/音频/音乐由本包的 Node 服务通过宿主连接器 +(`context.hostConnector`)直接调用,文本模式走宿主 Agent——跑的都是你的 MiniMax 模型: +不需要配 API Key、不存凭据,宿主掌握凭据与出网。 + +## 能做什么 + +- **无限画布** — 平移缩放的节点图。节点是 DOM 元素,连线是一层 SVG,整个场景挂在 + 一个 CSS transform 下。网格是 CSS 背景,滚动和缩放都零重绘。 +- **两族节点、四种数据、七种节点** — **输入**节点装着要送进去的东西(文本 / 图片 / 视频 / 音频, + 类型在卡片上选),**生成**节点产出其中一种;还有**结果**节点(生成完自己长出来)和 + **分组**框。线上流动的数据种类就 `text` / `image` / `video` / `audio` 四种: + **风格预设和提示词扩写都是文本**,所以它们不是新的种类——风格就是一个用「品类模板」 + 填好的文本输入。所以画布上一共 **7 种节点**:输入 1 种(装什么在卡片上选)、生成 4 种, + 再加结果和分组。两个添加菜单(空白处右键、 + 右上角「添加节点」)列的就是能加的 6 种(结果节点自己长出来), + **和卡片读的是同一张表**,菜单和画布不会对不上。 + **模式**是同一种数据的另一种做法,放在卡片上选:文本节点可以写文本,也可以把一句话扩写成 + 提示词(提示词增强)、读一张图(图生文)、把一条视频反推成提示词(视频生文);音频节点可以 + 配音,也可以写歌(文生音乐)。所以模式不进菜单。 + 而且模式说明了这个节点「要什么」,所以**左侧的入端口由模式决定**:文生图等一个文本, + 图生图再等一张图。 + 所有形态写在**一张表**里,页面和服务端读的是同一张。 +- **视频生文把视频反推成提示词** — 把视频接进文本节点的「视频生文」模式,它会写出能生成这条 + 视频的提示词:provider 的视频理解逐镜头读视频,文本模型把它看到的东西写成一条能直接用的 + 提示词。**文本模型自己看不见视频**,所以运行时先把视频读一遍,再把描述交给它。 +- **图生文对图片做同一件事** — provider 的图像理解先读这张图,文本模型再写描述。 + 少了这一步,文本模型只会拿到一个 URL,而它打不开 URL。 +- **文本框就是文本的入口** — 生成节点卡片上有一个文本框,文本就进这里。文本**不再单独占一个 + 端点**:文本线直接连到文本框上,插槽条里也不再列文本——一件事只在一处。 + 文本线接进来后,文本框变灰、不能输入(文本已经是上游的了);你自己写的提示词**保留在节点里** + (断开这条线就回来),但**这一次不发送**,所以卡片上看到的就是模型收到的。 +- **连线即数据流** — 上游节点的产物会作为「参考材料」注入下游提示词。环路会被拒绝。 +- **节点内嵌提示词编辑器** — 用的是 contentEditable 而不是 textarea,所以 `@` 引用 + 可以直接嵌在正文里。中文输入法组合输入单独处理,编辑器里的按键不会触发画布快捷键。 +- **`@` 引用芯片** — 引用是带缩略图的不可编辑芯片。引用的素材一旦消失,芯片会变成 + **孤儿态**并明确提示。 +- **按厂商编译的引用语法** — mention 在内部编码为 `[[cw:kind:nodeId]]`,再按厂商家族 + (kling / minimax / wanxiang / bytedance)编译成该厂商自己的引用写法,同时把素材 URL + 收进独立数组一起交给模型。 +- **随模型能力变化的参数** — 参数面随生成类型和**所选模型**走:图片有比例和清晰度; + 视频再加时长和「续写上一镜头」;音频再加音色和语音情绪。模型不支持的取值会被自动 + 吸附到它支持的档位上。比例这一栏会**标出画面朝哪边**(16:9 横屏、9:16 竖屏、1:1 方形), + 但发给 provider 的值还是那个比例本身。 +- **引用上限由服务端强制** — 不信任页面。超过模型上限、或超过全局 12 个引用的上限, + 都会在任何 run 启动**之前**就被拒绝。 +- **能力清单面板** — 展示每种生成模式的实时参数面,以及宿主 matrix provider 的**全部**连接器 + 工具和它们的授予状态。provider 提供的工具比本 App 用到的多,所以每一行会标出 + **「未授予」**(宿主没给)和**「本 app 不用」**(本 App 没有功能走它)——面板不替谁吹牛。 +- **本机媒体工具** — 14 个跑在本机的变换:抽帧 / 宫格抽帧 / 片段剪辑 / 视频剪裁 / + 自动分段 / 音画分离 / 合并视频,外加字幕擦除 / 视频去水印 / 背景移除,以及 + 智能抠图 / 变清晰 / 无损改尺寸 / 证件照。它们是本机变换而不是厂商能力,所以需要 + 本机有 `ffmpeg`;没有时按钮会保持禁用并说明缺什么。 + 参考实现的四个图片工具是**有意不做的**:拼图(要好几张图,不是一张图的快捷处理)、 + 海报(排版太粗)、去水印(得先框出要去掉的矩形)、服装换色(整图色相偏移,背景 + 跟着一起变)。结果节点上的工具是「对这一张图的快捷处理」,所以它必须只对这一张 + 图有效。 +- **视频理解** — 视频拆解 / 爆款分析走 provider 自己的视频理解能力,产出文本节点。 +- **人在环内** — Agent 请求工具权限时,审批卡直接长在节点上。只能选「允许一次」, + 且必须由用户先做决定;页面绝不会替用户自动批准。 +- **逐次取消** — 左下角运行进度面板里,每一次运行都有独立的「取消」。取消只停这一 + 次:其它运行照跑,它背后的 Agent 调用通过自己的中止控制器拆掉,节点也立刻空出来 + 可以重跑。 +- **在对话里打开** — 走过 Agent 的运行可以一键在宿主的对话侧栏里打开那段会话,直接 + 在能读到的对话里接着聊。生成节点上也留了「对话」按钮,所以跑完的会话同样回得去。 + 节点上的「新会话」则把这段会话丢掉,下次生成从头开一段新的。 +- **产物落子节点** — 生成完成后会新建一个产物类型的节点并自动连线,画布始终是图 + 而不是列表。 +- **三种角色、三类节点** — **输入**节点装要送进去的东西(文字 + 本地文件), + **生成**节点管提示词、模型和参数,**结果**节点在生成完成时自动长在它旁边。 + 生成节点自己不留结果,所以链条从左到右读:输入 → 生成 → 结果 → 生成 → 结果。 +- **本地素材** — 输入节点可以直接选本机的图片、视频或音频。文件存在运行时自己的数据目录里 + 并从那里提供;当它要作为生成的参考图时,会先上传给 provider(provider 够不到回环地址)。 +- **拉到空白处出筛选器** — 从节点右侧出线口拉到空白处松开,会弹出可用节点列表(只列真正 + 吃得下这个上游的节点类型),选中即在该处建好并自动连线。 +- **端口跟着模式走** — 生成节点按它选的模型给出它吃得下的几种输入端口:图片模型给 + 文本 + 图片两个口,支持多模态参考的视频模型给文本/图片/视频/音频四个口。从入端口 + 往空白处一拉,就地建出这个口需要的输入节点并自动连线。**线本身也从端口出、到端口止**, + 多个端口的节点就是一根线对一个口。 +- **每张卡片都声明自己的接口**(ComfyUI 的做法)— 卡片头部下面那条插槽条直接写出 + **它吃什么、吐什么**:左边「入」是能接进来的数据种类,右边「出」是它产出的种类。 + 种类跟着所选模型走,所以卡片不会给出一个永远填不上的口。**还空着的入槽是空心环, + 旁边写着「待接入」**;接上了就是实心点,写「已接入」;节点自己已经写了提示词, + 文本槽就说「已就绪」。不用碰任何东西,就知道还缺什么没接。 +- **模式跟着输入走** — 图片节点接上图片输入,就自动变成「图生图」;只有 + 真需要这种输入的选项,接上了才出现。 +- **提示词增强节点** — 一句话 + 一个提示词模板,经文本模型扩写成符合模板要求的 + 提示词。模板可以下拉选预设,也可以直接改写。预设里除了电商主图、短视频分镜这些, + 还有五个 **MiniMax H3 的形状**(T2VA / I2VA / FL2VA / L2VA / 全参考六段)—— + 按 MiniMax 官方的提示词指南来:结构用英文写,台词保留原语言。 +- **上游生成的文本会接管提示词** — 上游接的是文本结果时,本节点的输入框变成只读面板, + 显示接过来的文本,因为已经没有东西可打了。 +- **结果节点落在链路中间** — 生成节点后面接的是另一个生成节点时,结果节点就建在两者 + 之间,连线也改走它,链条读起来是 生成 → 结果 → 生成,下一个节点读的就是这个结果。 +- **整条链自己跑** — 点链路末端节点的「生成」,会自动把上游还没出结果的生成节点按 + 从深到浅的顺序补齐;已经有结果的直接用,不会重复跑。同一步再跑一次是**替换**旧结果, + 不会堆出第二份。 +- **一次出多张** — **生成**按钮旁边有一个「数量」下拉(1/2/3/4/6/8)。选 4 就出 4 个版本。 + **宿主对一次连接器调用有 30 秒上限**,所以 4 张是 **4 次调用、每次一张**(每次各自有 30 秒窗口), + 不是一次请求里塞 4 个——那样会超时,而且超时的那次可能已经扣了费。 + **结果每个一张卡、并排排成一行**,方便对着比;其中一张失败,其余照样出得来,运行日志会说明少了哪张。 + 视频(一个片子一个任务)和文本(一轮一个回复)不批量,那两种模式下就没有这个下拉。 +- **慢调用一次只跑一个** — 30 秒的窗口是给**一次调用**的,所以和别的调用抢机器的那个才是会超时的。 + 于是慢调用(图片、语音、音乐、视频提交)之间会排队:同一张图的第二个角度等第一个角度跑完再开始, + 各自都拿得到完整窗口。**一次失败不会挡住后面的调用**。排队等了多久会写进运行日志,失败时也会写, + 「provider 慢」和「在等机器」这两件事不会混为一谈。 +- **卡片之间按卡片自己的尺寸留距离** — 一次跑出来的结果、宫格切分的每一块、工具产出的子节点, + 走的都是同一条规则:两张卡之间的距离 = 卡片自己的尺寸 + 一点余量。卡片自己的尺寸是页面量出来的 + (预览框跟着素材的比例走、工具栏会换行),所以一批卡片量好之后会再排一次。 + 六十像素就挨着的两张结果卡会互相盖住,盖住的那张图就看不见了。 +- **卡片不会压着另一张** — 新放的卡片压到了谁,就把谁往右边让开,那边有地方。 + 多角度的一个分支的结果本来会正压在旁边的分支上,现在让开的是分支。 +- **这台机器上已经没了的素材会被放在外面** — 节点上面的链条就是这次带的参考素材,而 + provider 那边同样读不到这台机器上已经不在的文件。运行会说清楚少了哪一份,然后带着 + 剩下的继续跑,而不是死在它上面。 +- **配色跟着系统** — 默认跟随系统的明暗,用户也可以自己选,选完会记住。 +- **Agent 能读画布** — 本 Mini App 对外提供一个托管 MCP 服务(`servers.mcp.json`), + Agent 因此有自己的工具:`canvas_list` 列出画布,`canvas_read` 读回节点、连线 + 和每个节点的插槽(和卡片上是同一套三态)。这两个工具**只读**。 +- **Agent 能改画布** — 另外五个工具能写画布:`canvas_add_node`、`canvas_connect`、 + `canvas_update_node`、`canvas_remove_node`、`canvas_run`。它们改的是用户看得见的东西, + 所以都不标只读,工具说明里也写明。 +- **模板是一条链的骨架** — 「模板」菜单里有一键铺好的**七条现成工作流**,按分组摆: + - **从素材出发** —— **图片复刻**、**视频模仿**:把素材先读回成一段提示词, + 再拿这段提示词去生成; + - **H3** —— 每种请求能带的图都有一条:T2VA(文字)、I2VA(首帧)、FL2VA(首帧+尾帧)、 + L2VA(尾帧)、全参考。每条都带一个**提示词增强**节点,按 H3 期待的结构写提示词 + (点名那张图的那一行由运行时补上)。 + + 这七条是**文件**(`miniapp/templates/*.json`),不是写死在代码里的:模板是数据, + 用户打开一条、改好、存成「我的模板」;要改预置的那几条,也是改文件,不用动代码。 + + 模板只有**步骤**(节点、连线、提示词、参数),**不带任何跑出来的结果,也不带素材** —— + 所以换个人、换台机器铺出来照样能用,不会像导入一张画布那样满屏裂图。铺完**不自动跑**:钱是你按「生成」 + 才开始花的。 +- **一个按钮、一个菜单,进两条路出两条路** — 工具栏上只留一个**「模板」**按钮,它打开的菜单里是: + - **我的模板** —— 你自己留着的链(收在运行时自己的模板文件夹里,文件没了它还在)。 + Agent 也看得见:`canvas_list_templates` 列出全部,`canvas_apply_template` 一键铺一条; + - **打开本地模板** —— 本机上现成的模板文件,由你自己挑; + - **保存为我的模板** —— 当前画布变成你自己的一条,起个名字(只留步骤,结果和素材不留); + - **保存到本地** —— 当前画布写成一个文件,存到你说的任意文件夹(下载文件夹、桌面、文档, + 或者你自己填的);文件由运行时写,页面告诉你落在哪儿。 + + 没有第三条进路,也没有第二种留法:指定文件夹找、给文件按「+」、再挑一个文件, + 本来就是同一件事的三种说法。 +- **在里挑中的会开一张新画布** —— 正在做的画布是进行中的活儿,不去动它 + (当前画布是空的就用它:上面没东西,丢了也不心疼)。 + 文件里只有步骤,没有结果也没有素材,所以换台机器打开也是完整的; + 一条链从哪张图起步,那个**空输入位**也跟着走,放的人自己往里放图。 +- **自己的模板能管** — 「我的模板」里每一条旁边都有删除按钮(删除前也是在页面里问, + 跟这里别的问题一样)。内置的那些不是你的,删不了。 +- **画布是工作台,它自己会存** — 改动后一秒左右自己落盘,页头画布名旁边那颗状态芯片说了算, + 所以工具栏上**没有「保存」按钮**:手动再存一次只是把页面已经在做的事做第二遍。 + 窗口关掉时那份本地草稿也跟着走,所以「刚改完就关」也不会丢。 + 真正值得**留住**的是**模板**(步骤)和**结果**(在资产库里,能存到本地)—— + 删画布之前,弹窗会先告诉你这些。 +- **这个包会告诉 Agent 怎么用它** — 附了一个 Skill(`skills/ai-node-canvas/SKILL.md`,中文): + 写清画布是干什么的、什么时候该先走**提示词增强**(H3 那几个模板带的是 provider 自己的提示词结构, + 手写一遍很容易丢掉)、引用编号怎么算、哪些链值得搭、哪些请求模型会拒。Agent 在任务是 + 「一条生成链」而不是「一句提示词」的时候会读到它。 +- **提示词增强节点带模板** — `canvas_add_node` 和 `canvas_update_node` 都接受 `templateId`, + 所以 Agent 建的提示词增强节点,用的是和页面上一样的模板(H3 那几个也在里面);页面和 + Agent 问的是同一条规则。 +- **页面仍然是唯一写者** — 画布活在页面的内存里,存储层会拒绝基于旧版本的写入。 + 所以 Agent 的改动**不会**在页面背后直接写进存储:页面在读画布之前先「认领」它, + 认领有效期间运行时就不写这张画布,改动以**动作**的形式交给页面;页面用运行时本来 + 也会用的那一条规则去应用(规则只有一份、在一个地方),然后回执,画布这才保存。 + 认领和写入走同一把锁,所以改动不可能落在「认领」和「读取」之间。没人打开画布时, + 运行时自己应用这个动作,它发起的运行结果也会自己落到画布上。 + **两种情况下画布长得一样**,因为放节点、画连线、落结果用的是同一个函数。 +- **一条颜色规则** — 画布只有两族颜色,含义固定: + - **内容色**按数据种类分:**文本=琥珀、图片=青绿、视频=蓝、音频=紫**。 + 节点头部、端口、插槽条、连线用同一个色,所以一条链可以只靠颜色读出来。 + - **交互色**统一是强调色(蓝):选中、焦点、hover 都是它,不跟内容色混。 + - 端口按数据种类上色;**输出是实心圆、输入是空心环,接上以后才变实心** —— 一眼看出哪些口还空着。 + - 选中一个节点时,**周围的节点和线保持满亮,其余退到背景里**,视线自然落在链上。 + - 能力清单面板里有一份「数据色」图例。 +- **状态不靠颜色说** — 选中的节点会显出一个「已选」标识,它的可访问名里也带着「(已选)」; + 节点本身是一块有名字的分组,提示词编辑器有自己的名字,跑进度面板是 `role="status"`, + 出错的横幅是 `role="alert"`。 +- **卡片跟着素材走** — 生成出的图片/视频按它自己的比例预览,卡片按预览调整尺寸, + 不再把素材硬塞进固定画框。9:16 的竖图得到窄而高的卡片,16:9 的横图得到扁卡片。 + 手动拖过尺寸的卡片会保留用户定的宽度。 +- **撤销 / 重做** — 快照式历史,覆盖节点、连线和视口。`Ctrl+Z` / `Ctrl+Shift+Z` / + `Ctrl+Y`。 +- **多选** — `Shift`+拖拽框选,`Shift`+点击增减选择,`Ctrl+C` / `Ctrl+V` 复制粘贴 + (同时写系统剪贴板),拖动时整组一起移动。 +- **小地图** — 实时显示全图与当前视口,点击可跳转。 +- **草稿恢复** — 画布会同步一份本地草稿;重新打开时若发现更新的草稿会提示恢复。 +- **本地持久化** — 画布保存在宿主的私有数据目录,可列出、打开、重命名、复制、删除。 + 改动自己会存,不用手动保存;关页面前最后一点改动也会先落进本地草稿。 +- **页面自己问** — 新建、重命名、删除三件事都在页面里问:名字直接打在页头的输入框里, + 删除由页面自己的确认框确认。原生 `prompt()` / `confirm()` 是独立系统窗口,在 MiniApp + 内嵌浏览器里根本不弹——那样问出去的问题没人回答,按钮看上去就像没反应。 + +## 连线语义 + +连线决定「素材怎么走」,也决定「节点接什么」。 + +生成节点左边能插几个端点、端点是什么种类,**由它当前的模式决定**(模式就是卡片上的「模式」 +下拉:文生图 / 图生图……)。所以: + +- **接一条新线**:如果这条线带的种类(文本 / 图片 / 视频 / 音频)当前的模式接不住,下游节点 + 会**自动切到接得住的模式**——而且是那种「既接得住新线、又接得住已经连上的所有线」的模式。 + 接了图 → 文生图变图生图。没有模式接得住时,这条线被拒绝,并说明 + 该切到哪个模式。 +- **手动切模式**:新的模式接不住的连线就没有端点可落,会被去掉,并提示去掉了几条。 +- 画布上的规则只有一份:手动拖的线和 Agent 通过 MCP 要的线,走的是同一条规则,落点一样。 + +被拒绝的连线只有四种:自己连自己、同一条线连两次、会形成环路的连线,以及**下游没有任何模式 +接得住这条线的种类**。 + +### 一张图变成视频有几种做法 + +provider 有**几种不同的请求**把素材变成视频,不是同一个请求换几个参数: + +| 请求 | 送什么 | 谁会跑 | +| --- | --- | --- | +| 首帧 | `input_image` + `reference_type: first_frame` | 三个模型都会 | +| 首尾帧 | 首帧 + `last_frame_image` | H3、H3-Max | +| 末帧 | `input_image` + `reference_type: last_frame` | H3、H3-Max | +| 参考集合 | `reference_images` / `reference_videos` / `reference_audios` | 只有 H3 | + +所以**模式说的是"要哪种请求",模型说的是"能跑哪几种请求"**,两件事分开。视频节点有 5 个模式, +每个模式要什么都是**精确的**——它接什么,请求里就有什么(文本走卡片上的文本框,不是端口): + +| 模式 | 它要什么 | +| --- | --- | +| 文生视频 | 文本框 | +| 图生视频·首帧 | 文本框、图片(首帧) | +| 图生视频·首尾帧 | 文本框、图片(首帧)、图片(尾帧) | +| 图生视频·末帧 | 文本框、图片(尾帧) | +| 参考生视频 | 文本框、图片、视频、音频 | + +- 首尾帧要**两张图**,所以卡片上有两个图片口,插槽条里写明**首帧**和**尾帧**;线落在哪个口上, + 请求里就是哪一张。**只接了一张图时它退回首帧请求**——provider 不会收一个"没有尾帧的首尾帧请求"。 +- 参考集合是**一个请求**,集合里可以同时放图、视频和音频:最多 9 图 + 3 视频 + 3 音频,总共 12 件。 + 所以一个模式把这三种都带上,卡片给的端口就是模型真收的那几种——H3 给图、视频、音频三个口, + 没有参考集合的模型干脆就没有这个模式。视频和音频参考跟图片参考是一回事:它们告诉模型跟着什么走, + 音轨还是模型自己生成。 +- provider 的三条硬规矩由运行时挡在前面(页面看不到):**参考集合至少要有一件**、 + **音频参考不能单独来**(要配一张图或一条视频),**集合总共最多 12 件**(9 图 + 3 视频 + 3 音频 + 是 15 件,会被拒)。不满足时运行直接拒绝并说明缺什么,不发一个 provider 一定会拒的请求。 +- 画幅这一栏跟的是**请求**不是模型:两个首帧请求恒为自适应,卡片上就不给这一栏;文生视频和参考 + 集合才可以选比例。 + +### 选项跟着模型走 + +视频节点的模型决定卡片上给什么,因为这三个视频模型不是一回事。下面的数字是从连接器那份 +**带条件的 schema** 里抄的,不是拍脑袋定的: + +| | MiniMax-H3 | MiniMax-H3-Max | MiniMax-Hailuo-2.3 | +| --- | --- | --- | --- | +| 能跑的模式 | 文生视频 · 首帧 · 首尾帧 · 末帧 · 参考 | 文生视频 · 首帧 · 首尾帧 · 末帧 | 文生视频 · 首帧 | +| 时长 | 4–15 秒 | 5–15 秒 | 6 秒或 10 秒 | +| 清晰度 | 768P、2K | 480P、768P | 768P;1080P 只配 6 秒 | +| 画幅 | 6 个比例 + `自适应`(跟随首帧) | 6 个比例 + `自适应`(H3 家族) | **没有**——它不是 H3 家族 | +| 参考素材 | 最多 9 图 / 3 视频 / 3 音频,总共 12 件 | 只有 1 张图,且是首帧 | 只有 1 张图,且是首帧 | +| 声音 | 自带同步音轨 | **成片静音**,要人声请用 H3 | 成片无声,要配音得另接语音合成 | +| 计费 | 账户余额 | 账户余额 | Token Plan | + +所以卡片跟着模型走:切到 H3-Max,**参考集合**(它没有参考集合)和音频口都不出现;切到 Hailuo, +只剩文生视频和首帧。Agent 通过 MCP 改模型时走的是同一条规则,所以谁改的,画布都一个样。 + +> H3 和 H3-Max 走**账户余额**,而账户没开「按量自动扣费」时 provider 会直接拒——本该有声的 +> 成片会变成「被拒」,而不是安静地出来。Hailuo 走 Token Plan,但它是静音的。 + +> 卡片上**没有**声音开关:以前有两个勾(同步配音、允许背景音),但视频运行是直连连接器的, +> 那两个勾落到的地方(Agent 的提示词)根本不在这条路上——勾了等于没勾。声音现在只由两件事决定: +> **模式**说请求里带不带音频参考,**模型**说成片自带不带音轨(写在模型下拉的说明里)。 + +### 文本框就是文本的入口 + +生成节点卡片上有一个文本框,文本就从这里进。文本**没有自己的端点**:文本线落在文本框上,不是 +落在卡片边缘;插槽条里也不再列文本。一件事只在一处。 + +文本线接进来之后,文本框变灰、不能输入——文本已经是上游的了。你自己写的提示词**保留在节点里** +(断开这条线就回来,可以继续改),但**这一次不发送**;而交出文本的那个上游也不再作为参考素材 +重复出现。所以卡片和实际运行永远一致:**框里是什么,模型收到的就是什么**。 + +连线「实际做了什么」取决于两端是什么: + +| 上游 | 下游 | 效果 | +| --- | --- | --- | +| `文本`(输入节点、文本结果、跑过的文本节点)| 生成节点 | 文本**接管**下游的文本框:框变灰显示上游文本、不可输入;上游文本就是这次的提示词 | +| `图片`/`视频`/`音频` | `文本` | 产物作为「参考素材」块注入 Agent 回合 | +| `图片`/`视频`/`音频` | `图片`/`视频`/`音频`/`音乐` | 素材**顺着连线**发给 provider 作参考;提示词里的 @ 说明提示词在讲它的哪一部分,且该类型以 @ 的清单为准 | +| `分组` | 任意 | 分组只是容器不是数据源,不会把内容带进下游 | + +模型的引用上限照旧生效(最多 9 图 / 3 视频 / 3 音频),且由服务端在 run 启动**之前** +校验:超限会明确报错,而不是到 provider 那边才失败。 + +## 安装 + +把整个 `ai-node-canvas` 目录(含隐藏的 `.minimax-plugin` 目录)复制到 MiniMax Code +的数据目录: + +```text +仓库: ai-node-canvas/ +装到: /plugins/ai-node-canvas/ +清单: /plugins/ai-node-canvas/.minimax-plugin/plugin.json +``` + +默认数据目录是 `~/.minimax`,所以一般装到 `~/.minimax/plugins/ai-node-canvas/`。 + +重启 MiniMax Code,确认插件已被识别并启用,然后对 Agent 说「打开节点画布」,或者 +从插件列表里直接打开。 + +## 用法 + +1. 双击画布空白处弹出节点面板,或点「添加节点」。 +2. 选中节点,写提示词、选模型。MiniMax 模型排在最前面并带星标。 +3. 在提示词里敲 `@` 就能引用上游节点的产物。 +4. 拖节点右侧的小圆点到另一个节点上即可连线。 +5. 点「生成」,上游产物自动流进下游提示词,结果落到新的子节点。 +6. Agent 需要工具权限时,在节点上回应审批卡。 + +### 快捷键 + +| 按键 | 作用 | +| --- | --- | +| `Ctrl+Z` / `Ctrl+Shift+Z` / `Ctrl+Y` | 撤销 / 重做 | +| `Ctrl+C` / `Ctrl+V` | 复制 / 粘贴选中节点 | +| `Delete` / `Backspace` | 删除选中 | +| `Shift`+拖拽 | 框选 | +| `Shift`+点击 | 增减单个节点选择 | +| `Escape` | 依次关闭:引用菜单 → 面板 → 菜单 → 节点面板 → 取消选择 | + +## Test 环境 + +- MiniMax Code 3.1.2 on Windows 11(Node 22) +- MiniMax Code 3.1.2 on macOS 15 +- 需要支持 MiniApp 与 Agent 回合的桌面端客户端。 + +## 本机媒体工具 + +14 个媒体工具都是本机变换,所以需要本机有 `ffmpeg` 与 `ffprobe`。查找顺序是 +`PATH` → WinGet 的 links 与 package 目录 → 常见安装位置;`NXC_FFMPEG` 可覆盖。 +二进制不随包分发:打包一个 ffmpeg 会远远超出包的体积上限。 + +没有 ffmpeg 时,工具按钮保持禁用并说明缺什么,`/api/media/status` 也如实汇报。 +其余功能(各种生成模式、画布、文本类工具)不依赖 ffmpeg。 + +**装着视频的卡片不挂工具栏**:成片是交付物,不是半成品,卡片下面挂十几个按钮反而是 +噪音。卡片下面那条空白的工具条也不留 —— 没有工具就没有工具条。 +视频工具仍然留在运行时里,Agent 依然能通过 MCP 调用它们。 + +### 在浏览器里画的那两个工具 + +**宫格切分** 把一张图切成网格,**剪裁** 保留用户拖出来的部分——这两个工具在页面里 +做,不需要 ffmpeg,也不花额度。 + +页面画出结果后把字节交给运行时,运行时把它发布进 assets 目录,和别的产物一样。 +所以结果是**一个真文件**:能存到用户电脑上、会出现在资产库里、下一个工具也能接着读。 +如果留在画布里当 data URL,它就只活在画布文档里。**剪裁** 会另外留下一张图,原图 +原地不动。 + +### 问模型的那两个工具 + +**多角度** 和 **打光** 走 Agent。多角度可以选好几个角度,但一次生成没法同时给出两个 +角度,所以**每个角度跑一次**,每次的提示词只写自己那个角度。每次跑都是一次生成, +所以会花用户额度——面板会在你按下按钮之前告诉你这次会跑几次。 + +### 一条链走到底 + +每个工具留下的都是**结果节点**——不管是哪种做法。只有结果节点带工具栏,所以工具 +处理出来的图还能接着处理:智能抠图 → 变清晰 → 宫格切分 是一条链,不是三张各不 +相干的图。 + +## 数据与访问 + +- **连接器用来做什么** — `matrix` 连接器只用来生成媒体:图片、视频、语音、音乐, + 以及读一段视频做理解。凭据与出网都在宿主手里,本包不碰任何自己的账号信息。 +- **读取的文件** — 只有本包自己的页面资源,从 `context.pluginRoot` 读取 + (`miniapp/client/index.html`、`app.js`、`mention.mjs`、`tools.mjs`、`forms.mjs`、 + `actions.mjs`、`styles.css`)。不读取任何宿主文件。 +- **写入的文件** — 画布文档写在 `context.dataDir/canvases/*.json`,采用原子写 + (同目录临时文件再 rename);媒体工具另写工作文件与产物,分别在 + `context.dataDir/work/` 和 `context.dataDir/assets/`,上传的文件放在 + `context.dataDir/uploads/`。 +- **唯一写到文件夹外的地方** — 节点上的「保存」、资产库里的保存,以及菜单里的 + 「保存到本地」,都会把文件写进用户的**下载文件夹**(保存到本地写到你指的那个文件夹), + 名字就是节点或画布带的那一个。只有用户点了才写,页面也会告诉你文件落在哪儿; + 已经在的文件不会被覆盖,副本会带序号。走这条路,是因为浏览器自己发起的下载 + 在这个宿主里落不到它该有的名字上(文件停在临时名上)。 + 指的文件夹必须是真的在那儿的,文件名必须是一个干净的名字。 +- **读的是用户亲手给的那个文件** — 只读「打开本地模板」里用户挑中的那一个文件, + 而且只认**按「读模板」那一条规则**算得上模板的;不是模板的文件会被拒,也根本不会被列出来。 + 不扫用户的任何文件夹:文件是用户递过来的。 +- **网络主机** — 生成 provider 在 `get_asset_url` 里返回的素材所在主机,以及 + provider 的上传命令返回临时链接的那个主机,没有别的。Node 服务会把生成好的 + 产物下载一份存本地(这样画布里的链接不会过期),本地文件要作为生成参考时会先 + 上传给 provider(provider 够不到回环地址)。其余请求(图片、视频、音频、音乐、 + 音色清单、视频理解)全部走宿主连接器(`context.hostConnector`),凭据与出网由宿主掌握。 +- **上传的两步** — provider 的上传命令不收本地路径(provider 读不到本机磁盘): + 它收文件名与媒体类型,回一份预签名表单;运行时再把字节以 `multipart/form-data` + POST 到那份表单指向的主机(表单字段原样带上,文件放在 `file_field_name` 指定的那一项)。 + 拿到的 `temp_url` 才是 provider 能取到的地址。表单指向的主机必须是 https + (回环地址除外),请求由运行时自己发出,**不走**宿主连接器。 +- **连接器调用的时限** — 宿主对一次连接器调用有超时上限(默认 30 秒),超时的调用会被 + 中止并报「结果未知」:调用可能已经发出去了,也可能已经扣费,所以**不能自动重试**。 + 慢的生成(带参考图的图片生成最容易碰到)因此可能失败一次,失败信息里会写明是哪一个 + 工具、跑了多少秒。视频走「提交 + 轮询」,不受这个时限影响。 +- **启动的进程** — 只有 `ffmpeg` 与 `ffprobe`。两者都在本机查找,调用时用参数数组、 + 绝不经过 shell,所以路径或参数里的任何字符都不会被当成 shell 语法。具体命令: + - `ffmpeg -hide_banner -loglevel error -y <输入> …` — 每个媒体工具一次调用: + 取帧(`-ss`、`-frames:v 1`)、宫格拼图(`fps=…,tile=CxR`)、片段剪辑 + (`-ss`、`-to`、libx264/aac)、画幅裁剪(`crop=W:H:X:Y`)、镜头检测 + (`select='gt(scene,…)',showinfo`)、音画分离(`-an` / `-vn`)、合并 + (`-f concat -safe 0 -i <清单>`)、抠底(`chromakey=0xRRGGBB`)、去水印 + (`delogo=x:y:w:h`),以及图片侧的 `colorkey`、`scale`、`pad`、`crop`、 + `unsharp`、`hqdn3d`。 + - `ffprobe -v error -print_format json -show_streams -show_format <输入>` — + 读时长、分辨率、帧率,以及有没有音轨。 + 所有进入 filter 字符串的参数都先校验:数字强制转换并夹紧,选项值一律取自固定 + 白名单,页面传来的值不可能变成 filter 语法。派生的子进程会被跟踪,运行时销毁时 + 一并结束。 +- **凭据** — 不读取、不存储、不转发。宿主负责回环端口鉴权,并在请求到达本包前 + 摘掉它的凭据头。 +- **托管 MCP 服务** — `servers.mcp.json` 声明了一个 Streamable HTTP 服务, + 由 `miniapp.json` 挂在 `/mcp/canvas` 上;宿主会改写它的地址,并像端口其余部分 + 一样鉴权。这个服务说的是无状态 JSON-RPC(`initialize`、`ping`、`tools/list`、 + `tools/call`),提供三个读工具和六个写工具: + - `canvas_list` — 列出画布:id、标题、节点数、最后修改时间。 + - `canvas_read` — 读一张画布:节点(类型、标题、状态、模型、参数、失败时的原因)、 + 连线,以及每个节点声明的插槽和它在卡片上的状态(`filled` / `ready` / `open` / `out`); + 带名字的口(首帧 / 尾帧)会说自己是谁,两张图片口不会看成一张。 + - `canvas_list_templates` — 列出现成工作流和用户自己的模板。 + - `canvas_add_node` / `canvas_connect` / `canvas_update_node` / + `canvas_apply_template` / `canvas_remove_node` / `canvas_run` — 改画布:加节点、连线、 + 改节点、把一条工作流铺上去、删节点、跑一次。 + + 读工具**只读** `context.dataDir/canvases/` 下的画布文档:不写、不发起运行、 + 不调 provider,所以 Agent 看画布时不会撞上用户正在做的事。写工具走的是**动作**通道: + 页面开着就交给页面应用并回执,没人开就运行时自己应用——**不会**在页面背后直接改存储。 + 这个端点不依赖任何第三方包——依赖目录不随包发布,协议是手写的。 +- **连接器授权** — manifest 在 `hostConnectorAccess` 里声明了 `matrix` provider,由宿主 + 在安装时授予。本包**确实会直接调用**它: + - 图片/视频/音频/音乐由本包的 Node 服务通过 `context.hostConnector` 直接调用 + (`generate_image` / `submit_video_generation` / `query_video_generation` / + `synthesize_speech` / `batch_text_to_music`),素材地址再用 `get_asset_url` 换成 + 可显示的短时效 URL(节点的 `nodeId` 一并落盘,链接过期可续)。 + - 音色下拉也是活数据:`/api/voices` 在处理请求时读一份真实的 TTS 音色清单。 + - 视频拆解 / 爆款分析用 `videos_understand` 读视频。素材由 provider 自己去拉, + 所以这两个工具对「生成出来的视频」可用;本 MiniApp 本机加工出来的文件挂在 + 回环地址上,provider 拉不到。 + - 调用路径不改变计费口径:同一条连接器、同一个账号,选哪个模型决定走账户余额 + 还是 Token Plan(H3/H3-Max 走账户余额,Hailuo-2.3 走 Token Plan)。 + - 文本模式没有对应的连接器工具,仍走 Agent 回合。 +- **Agent 回合** — 有,但只用于文本模式与提示词类的创作工作。Agent + 保留完整能力:文件、Shell、Skills、MCP,也包括本 Mini App 自己的工具,因此它可以 + 读写文件、执行命令、调用 MCP 工具。 + 权限请求会呈现在节点上,只有用户先做了决定才会回 `allowOnce`;页面不会自动批准。 + 服务端会把任何非 `allowOnce` 的决策一律降级为 `deny`。 + 来自页面、文件或网络的文本一律当作数据,不当作指令。 +- **状态位置** — `context.dataDir`,宿主为本插件创建并拥有的私有目录。位置视为不透明, + 版本升级可能变化,请不要写死。浏览器侧另存一份本地草稿用于崩溃恢复。 diff --git a/plugins/antianqi/ai-node-canvas/icon-dark.png b/plugins/antianqi/ai-node-canvas/icon-dark.png new file mode 100644 index 0000000..84c0aa8 Binary files /dev/null and b/plugins/antianqi/ai-node-canvas/icon-dark.png differ diff --git a/plugins/antianqi/ai-node-canvas/icon.png b/plugins/antianqi/ai-node-canvas/icon.png new file mode 100644 index 0000000..92c160d Binary files /dev/null and b/plugins/antianqi/ai-node-canvas/icon.png differ diff --git a/plugins/antianqi/ai-node-canvas/miniapp/client/actions.mjs b/plugins/antianqi/ai-node-canvas/miniapp/client/actions.mjs new file mode 100644 index 0000000..ac576d8 --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/miniapp/client/actions.mjs @@ -0,0 +1,1433 @@ +// Actions: how the Agent asks the canvas to change. +// +// The page is the only writer of a canvas: it holds the document in memory and +// saves it, and the store refuses a write that is based on an older version. +// An action that comes from the Agent is therefore never written to the stored +// document behind the page's back — it is applied by the page, and by the +// runtime only while no page holds the canvas. +// +// Every rule here is pure, so the page and the runtime apply the same one: +// what a node looks like, what a run takes with it, and where a result lands. +// +// The kinds of data come from the node table in `forms.mjs`, never from a +// second list written down here. + +import { CREATABLE_TYPES, DATA_KINDS, kindLabel, kindOf, modeLabelOf, modeOf, modesFor, modesOf, pathOf, roleOf, runModeOf, slotsOf } from './forms.mjs'; +import { batchCount, fitParamsToCaps, framePorts, assignPorts, directLines, planBatchPlacement, portKey, pushAside, takenPorts, upstreamNodes } from './tools.mjs'; + +/** The actions an Agent may ask for. */ +export const ACTION_TYPES = ['add_node', 'connect', 'remove_node', 'update_node', 'run', 'apply_template']; + +/** How many nodes one canvas may hold. */ +export const MAX_NODES = 300; + +/** How many connections one canvas may hold. */ +export const MAX_CONNECTIONS = 1000; + +/** The longest prompt a node may carry. */ +const MAX_PROMPT = 20000; + +const IDLE_OUTPUT = { + status: 'idle', + text: '', + media: [], + error: '', + posterUrl: '', + durationMs: 0, + width: 0, + height: 0, + missing: false, + updatedAt: '', +}; + +/** A fresh id, when the caller has no rule of its own. */ +function fallbackId(prefix) { + return `${prefix}_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 8)}`; +} + +/** + * The materials a prompt names. + * + * A mention points at a node. The material behind it is a reference of the + * node that names it, and the name it gets in the prompt is its place in the + * request — so the two have to be the same picture. + */ +export function mentionedMaterials(canvas, node) { + const segments = node?.metadata?.segments ?? []; + const out = []; + for (const segment of segments) { + if (segment?.type !== 'mention') continue; + const source = (canvas?.nodes ?? []).find((item) => item.id === segment.nodeId); + if (!source) continue; + const media = source.metadata?.output?.media ?? []; + if (media.length === 0) continue; + const kind = media[0]?.kind ?? 'image'; + if (out.some((item) => item.id === source.id)) continue; + out.push({ + id: source.id, + kind, + title: source.title, + media, + node: source, + }); + } + return out; +} + +/** + * The nodes a node feeds. + */ +function childrenOf(canvas, nodeId) { + const out = []; + for (const conn of canvas?.connections ?? []) { + if (conn.fromNodeId !== nodeId) continue; + const child = (canvas?.nodes ?? []).find((item) => item.id === conn.toNodeId); + if (child) out.push(child); + } + return out; +} + +/** + * Does this node already have that material? + * + * The chain above it counts: an input three hops up still reaches the node. + */ +function hasMaterial(canvas, node, materialId) { + if (node?.id === materialId) return true; + return upstreamNodes(canvas, node?.id).some((item) => item.id === materialId); +} + +/** + * The holder of a material: an input node that carries it. + * + * The material may come from anywhere on the canvas, and the node that + * generates from it needs it as an input of its own. A holder keeps the flow + * independent of where the material came from, and it is an input node, so + * the chain never runs it again. + */ +function holderOf(canvas, material) { + // A holder is its own holder: once the prompt names the holder, the rule + // must not make a second one for it. + if (material.node?.type === 'input' && material.node?.metadata?.materialFrom) { + return material.node; + } + return (canvas?.nodes ?? []).find((node) => ( + node.type === 'input' + && node.metadata?.materialFrom === material.id + )) ?? null; +} + +/** + * Wire the material a prompt names into the nodes that need it. + * + * A mention says "this prompt is about that material". The node that writes + * the prompt looks at it, and the node that generates from the prompt has to + * receive it as an input of its own — otherwise the prompt names a picture + * the request never sends. So the material is wired in: an input node holds + * it, and a line carries it to every node that takes that kind and does not + * have it yet. + * + * The rule is pure, so the page and the runtime read the same one, and it + * says nothing when there is nothing to do. + */ +export function syncMentions(canvas, nodeId, context = {}) { + const doc = canvas ?? { nodes: [], connections: [] }; + const nodes = [...(doc.nodes ?? [])]; + const connections = [...(doc.connections ?? [])]; + const ctx = { idFor: fallbackId, ...context }; + const index = nodes.findIndex((item) => item.id === nodeId); + if (index < 0) return { canvas: doc, added: [], connected: [] }; + const source = nodes[index]; + + const added = []; + const connected = []; + // The materials the prompt names, and the node that holds each one. + const materials = mentionedMaterials(doc, source); + if (materials.length === 0) return { canvas: doc, added, connected }; + + const mentionRewrite = new Map(); + + for (const material of materials) { + // The nodes that want this material: the one that names it (when its own + // mode takes this kind) and the ones it feeds. + const targets = []; + if ((takenPorts(source) ?? []).some((port) => port.kind === material.kind)) { + targets.push(source); + } + for (const child of childrenOf(doc, source.id)) { + if ((takenPorts(child) ?? []).some((port) => port.kind === material.kind)) { + targets.push(child); + } + } + const wanted = targets.filter((target) => !hasMaterial(doc, target, material.id)); + if (wanted.length === 0) continue; + + // The holder: one per material, so the same picture is not copied twice. + let holder = holderOf(doc, material); + if (!holder) { + if (nodes.length >= MAX_NODES) continue; + const id = String(ctx.idFor('n')); + const size = ctx.sizes?.input ?? { width: 300, height: 300, title: '输入' }; + holder = { + id, + type: 'input', + title: material.title, + position: { + x: Math.round((source.position?.x ?? 120) - 340), + y: Math.round(source.position?.y ?? 120), + }, + width: Math.round(size.width ?? 300), + height: Math.round(size.height ?? 300), + metadata: { + templateId: 'custom', + templateVars: {}, + segments: [], + prompt: '', + mentions: [], + model: null, + ratio: null, + resolution: null, + duration: null, + voice: null, + emotion: null, + sessionId: null, + // What the card holds: an input node holds what the user put in it. + kind: material.kind, + mode: null, + templateText: '', + // The node the material came from, so the holder is found again + // instead of a second one being made for the same picture. + materialFrom: material.id, + output: { ...IDLE_OUTPUT, status: 'success', media: material.media }, + }, + }; + nodes.push(holder); + added.push(holder); + } + + for (const target of wanted) { + if (connections.length >= MAX_CONNECTIONS) break; + if (connections.some((conn) => conn.fromNodeId === holder.id && conn.toNodeId === target.id)) continue; + const port = planPort({ ...doc, nodes, connections }, target, material.kind, ''); + if (!port.ok) continue; + // The line says which port it fills. A mode that takes the kind once + // has that port, and the line has to say so. + const kindPorts = (takenPorts(target) ?? []).filter((item) => item.kind === material.kind); + const role = port.role || (kindPorts.length === 1 ? kindPorts[0].role : ''); + connections.push({ + id: String(ctx.idFor('c')), + fromNodeId: holder.id, + toNodeId: target.id, + kind: material.kind, + role, + }); + connected.push({ from: holder.id, to: target.id, kind: material.kind, role }); + } + // The node that names the material now has it as an input of its own, so + // the prompt has to name that one: the number in the prompt is the place + // the picture has in the request. + const namesIt = targets.some((target) => target.id === source.id); + const holdsIt = connections.some((conn) => ( + conn.fromNodeId === holder.id && conn.toNodeId === source.id + )); + if (namesIt && holdsIt) mentionRewrite.set(material.id, holder.id); + } + + if (mentionRewrite.size > 0) { + const at = nodes.findIndex((item) => item.id === source.id); + const node = nodes[at]; + const segments = (node.metadata?.segments ?? []).map((segment) => ( + segment?.type === 'mention' && mentionRewrite.has(segment.nodeId) + ? { ...segment, nodeId: mentionRewrite.get(segment.nodeId) } + : segment + )); + nodes[at] = { + ...node, + metadata: { + ...node.metadata, + segments, + mentions: segments + .filter((segment) => segment.type === 'mention') + .map((segment) => ({ nodeId: segment.nodeId, kind: segment.kind })), + }, + }; + } + + return { + canvas: { ...doc, nodes, connections }, + added, + connected, + }; +} + +/** Whether a path leads from one node to another. */ +export function reaches(canvas, fromId, targetId) { + const connections = canvas?.connections ?? []; + const stack = [fromId]; + const seen = new Set(); + while (stack.length) { + const current = stack.pop(); + if (current === targetId) return true; + if (seen.has(current)) continue; + seen.add(current); + for (const conn of connections) { + if (conn.fromNodeId === current) stack.push(conn.toNodeId); + } + } + return false; +} + +/** The model a new generating node starts with. */ +function pickModel(context, type) { + const models = context.modelsByType?.[type] ?? []; + if (models.length) return models[0].id ?? models[0] ?? null; + return context.defaultModelKey ?? null; +} + +// ---------------------------------------------------------------- the contract +// +// A node's mode says what it takes, and the ports on its card are exactly that. +// So a line only has somewhere to land when the node takes the kind the line +// carries. Two changes have to obey this, and they are written down here once: +// +// - a line comes in: the node moves to a mode that takes it (and everything +// already connected), or the line is refused with a reason; +// - the mode changes: a line the new mode does not take has nowhere to land, +// so it goes, and the caller is told which ones went. +// +// The page and the runtime both apply these rules, so a change made by the +// Agent lands in the same shape as one made by hand. + +/** + * The kind a line carries. + * + * The kind travels with the source, not with the line: a line drawn before the + * source changed what it holds still carries what the source holds now. + */ +export function lineKind(canvas, conn) { + const source = (canvas?.nodes ?? []).find((node) => node.id === conn?.fromNodeId); + return source ? kindOf(source) : (conn?.kind ?? ''); +} + +/** + * The ports a node takes right now: the inputs of the mode it is in. + * + * A port is a kind and, when the mode takes the same kind twice, the role + * that says which one it is. 图生视频·首尾帧 takes two pictures — one for each + * end of the clip — and the role is what tells the two lines apart. + */ +export { assignPorts, directLines, portKey, takenPorts } from './tools.mjs'; + +/** + * The port a line lands on. + * + * Most modes take a kind once, and the line lands on that slot however many + * lines of that kind arrive. A mode that takes the same kind twice has a port + * for each: there the role says which port the line belongs to, and a port + * that is already filled refuses the line. + */ +export function planPort(canvas, node, kind, role = '') { + const ports = takenPorts(node).filter((port) => port.kind === kind); + if (ports.length < 2) return { ok: true, role: '', reason: '' }; + const filled = assignPorts(canvas, node); + // A named port is a slot of its own: the first frame is the first frame, + // and a second line that asks for it is refused instead of landing on it + // too and leaving the other port empty. + const wanted = role + ? ports.find((port) => port.role === role && !filled.get(portKey(port))) + : ports.find((port) => !filled.get(portKey(port))); + if (!wanted) return { ok: false, role: '', reason: '这个口已经接上了。' }; + return { ok: true, role: wanted.role, reason: '' }; +} + +/** + * The port each line into a node fills, by the line's id. + * + * The mode decides how many ports there are, so when the mode changes the + * lines are handed to the new ports and each one says which port it is on. + */ +export function stampPortRoles(canvas, node) { + const assigned = assignPorts(canvas, node); + const roles = new Map(); + for (const [key, line] of assigned) { + if (!line) continue; + const [, ...rest] = key.split(':'); + roles.set(line.connectionId, rest.join(':')); + } + return roles; +} + +/** + * The kinds that reach a node through the chain above it. + * + * The chain is the flow: material handed to a node is handed further down, so + * a kind that is two hops up still reaches this node. The card says the same + * thing — the slot strip reads the chain too — so the mode and the strip never + * disagree about what this node has. + */ +export function upstreamKinds(canvas, node) { + const kinds = new Set(); + for (const source of upstreamNodes(canvas, node?.id)) { + if (roleOf(source) === 'group') continue; + kinds.add(kindOf(source)); + } + return [...kinds]; +} + +/** + * The mode a node must be in to take one more kind of material. + * + * The node moves to the first mode that takes the new kind *and* everything + * already connected, so a line is never dropped behind the user's back. When no + * mode takes it, the line is refused and the reason says which mode would. + */ +export function planIncoming(canvas, node, kind) { + const modes = modesOf(node); + const current = modeOf(node); + if (!kind) return { modeId: current, error: '', changed: false }; + // A node that is not generated has no modes to move to: it takes the kind + // it shows — a result node takes what the generation handed it — and + // nothing else. + if (!modes.length) { + const shown = slotsOf(node).inputs.map((slot) => slot.kind); + if (shown.includes(kind)) return { modeId: current, error: '', changed: false }; + return { + modeId: current, + error: `这个节点不接${kindLabel(kind)}。`, + changed: false, + }; + } + // A mode takes a kind when it declares a port for it. + const takes = (mode) => (mode.ports ?? []).some((port) => port.kind === kind); + const currentMode = modes.find((mode) => mode.id === current); + if (currentMode && takes(currentMode)) { + // The mode already takes this kind. The port the line lands on is the + // role's business, and a port that is already filled refuses the line. + const port = planPort(canvas, node, kind); + if (!port.ok) return { modeId: current, error: port.reason, changed: false }; + return { modeId: current, error: '', changed: false }; + } + const wanted = [...new Set([...upstreamKinds(canvas, node), kind])].sort(); + const next = modes.find((mode) => ( + takes(mode) && wanted.every((need) => (mode.ports ?? []).some((port) => port.kind === need)) + )); + if (!next) { + const takers = modes + .filter(takes) + .map((mode) => mode.label); + return { + modeId: current, + error: takers.length + ? `这个节点现在不接${kindLabel(kind)}。切到「${takers[0]}」就能接。` + : `这个节点不接${kindLabel(kind)}。`, + changed: false, + }; + } + return { modeId: next.id, error: '', changed: next.id !== current }; +} + +/** + * The lines that have nowhere to land when a node changes its mode. + * + * The ports are the mode's inputs, so a line whose kind the new mode does not + * take cannot land on the card any more. + */ +export function linesDroppedByMode(canvas, node, modeId) { + const mode = modesOf(node).find((item) => item.id === modeId); + if (!mode) return []; + return (canvas?.connections ?? []).filter((conn) => ( + conn.toNodeId === node?.id + && !(mode.ports ?? []).some((port) => port.kind === lineKind(canvas, conn)) + )); +} + +/** + * Put the node on another model. + * + * The options follow the model: a value the model does not offer is snapped + * onto one it does, and a mode the model cannot run is left for one it can — + * the lines that mode cannot take go with it. The rule lives here, so the + * page, the Agent and the runtime all do the same thing. + */ +export function applyModel(canvas, node, modelKey, context = {}) { + const capsFor = typeof context.capsFor === 'function' ? context.capsFor : () => null; + const next = { ...node, metadata: { ...node.metadata, model: modelKey } }; + const caps = capsFor(next); + const from = modeOf(next); + if (!caps) return { node: next, dropped: [], mode: { from, to: from } }; + // The values the model does not offer are snapped onto ones it does. + Object.assign(next.metadata, fitParamsToCaps(next.metadata, caps)); + // And a mode the model cannot run is left for one it can. + const available = modesFor(next, caps); + if (available.length && !available.some((mode) => mode.id === from)) { + const to = available[0].id; + const dropped = linesDroppedByMode(canvas, next, to); + next.metadata.mode = to; + return { node: next, dropped, mode: { from, to } }; + } + return { node: next, dropped: [], mode: { from, to: from } }; +} + +/** + * The template a prompt node expands its sentence with. + * + * The page and the Agent make a prompt node the same way: it names one of the + * templates the palette offers, and the text of that template is what the + * expansion follows. A node that names none takes the first one, so it is + * useful before anything is typed — and a node the Agent made is as useful as + * one the user made. + * + * The text of the template follows the name, unless a text of its own comes + * with it. + */ +export function promptTemplateFor({ + mode = '', + templateId = '', + templateText, + templates = [], +} = {}) { + const list = Array.isArray(templates) ? templates : []; + const wanted = String(templateId ?? '').trim(); + const own = templateText === undefined ? null : String(templateText); + if (mode !== 'prompt') { + return { templateId: wanted || 'custom', templateText: own ?? '' }; + } + const found = list.find((item) => item.id === wanted); + if (found) return { templateId: found.id, templateText: own ?? String(found.text ?? '') }; + if (wanted) return { templateId: wanted, templateText: own ?? '' }; + const first = list[0]; + if (first) return { templateId: first.id, templateText: own ?? String(first.text ?? '') }; + return { templateId: 'custom', templateText: own ?? '' }; +} + +/** + * The template a workflow is made of. + * + * A template is a list of actions, and a node it makes is named by its + * order: `@1` is the first node, `@2` the second. The real ids are handed + * out when the template is applied, so the same template can be applied + * twice on one canvas. + */ +function templateRef(value, ids) { + const text = String(value ?? ''); + const match = /^@(\d+)$/.exec(text); + if (!match) return text; + const index = Number(match[1]) - 1; + return ids[index] ?? text; +} + +/** + * Put a template on a canvas. + * + * The template's steps are performed by the same rules that perform the + * Agent's actions, so a template lands on the board exactly as if the + * steps had been asked for one by one. The nodes are laid out from left to + * right in the order the template makes them, at the place the canvas is + * looking at. + */ +export function applyTemplate(canvas, template, context = {}) { + const doc = canvas ?? { nodes: [], connections: [] }; + const actions = Array.isArray(template?.actions) ? template.actions : []; + if (!actions.length) return { canvas: doc, error: '这个模板是空的' }; + // A template is a skeleton: it may make nodes and draw the lines between + // them. It may not take anything away — a template that could would tidy + // up a canvas it was dropped on, and the steps of a chain are not a licence + // to remove things. + for (const action of actions) { + const type = String(action?.type ?? ''); + if (!TEMPLATE_ACTION_TYPES.includes(type)) { + return { + canvas: doc, + error: `模板里有一步做不了的事:${type || '(没有说是什么)'}(模板只能:${TEMPLATE_ACTION_TYPES.join('、')})`, + }; + } + if (type === 'connect') { + const from = String(action?.args?.from ?? ''); + const to = String(action?.args?.to ?? ''); + if (!/^@\d+$/.test(from) || !/^@\d+$/.test(to)) { + return { canvas: doc, error: '模板里的线要指向模板自己做的节点(@1 这样的名字)' }; + } + } + } + const ctx = { idFor: fallbackId, ...context }; + const view = doc.viewport ?? { x: 80, y: 60 }; + const ids = []; + let next = doc; + for (const action of actions) { + const args = { ...(action.args ?? {}) }; + if (action.type === 'add_node') { + const id = args.id ?? ctx.idFor('n'); + ids.push(id); + args.id = id; + // The place is the beginning: the nodes of a template are a chain, so + // they are laid out in a row where the canvas is looking. + args.x = Number(args.x ?? view.x + ids.length * 380); + args.y = Number(args.y ?? view.y); + const added = applyAction(next, { type: 'add_node', args }, ctx); + if (added.error) return { canvas: next, error: added.error }; + next = added.canvas; + // A node is made with its kind and its text; the rest of the settings + // are set on it afterwards, by the same rule that changes a node. + const settings = {}; + for (const key of ['model', 'ratio', 'resolution', 'duration', 'count', 'voice', 'emotion']) { + if (args[key] !== undefined && args[key] !== null) settings[key] = args[key]; + } + if (Object.keys(settings).length) { + const set = applyAction(next, { type: 'update_node', args: { id, ...settings } }, ctx); + if (set.error) return { canvas: next, error: set.error }; + next = set.canvas; + } + continue; + } + // A line names the nodes by the order the template made them. + args.from = templateRef(args.from, ids); + args.to = templateRef(args.to, ids); + const applied = applyAction(next, { type: action.type, args }, ctx); + if (applied.error) return { canvas: next, error: applied.error }; + next = applied.canvas; + } + return { canvas: next, nodes: ids }; +} + +/** + * The skeleton of a canvas: what a template of it would be. + * + * A result node is what a run left behind, and an input node's material is + * the file somebody picked — neither belongs to the steps. What is left is + * the steps themselves: the nodes that generate, the lines between them and + * the prompts and parameters they carry. + */ +export function templateFromCanvas(canvas, { id = '', title = '' } = {}) { + const nodes = canvas?.nodes ?? []; + const connections = canvas?.connections ?? []; + const kept = []; + const index = new Map(); + for (const node of nodes) { + // A result is a result: what a run left behind is not a step. An input + // node IS a step — it is the place the material goes in — but the + // material itself stays behind: the template carries the empty holder, + // and the person who puts the template on their board puts their own file + // in it. + if (node.type === 'result') continue; + index.set(node.id, kept.length + 1); + kept.push(node); + } + if (!kept.length) return { id, title, actions: [] }; + const actions = []; + for (const node of kept) { + actions.push({ + type: 'add_node', + args: { + type: node.type, + title: node.title, + prompt: node.metadata?.prompt ?? '', + mode: node.metadata?.mode ?? undefined, + kind: node.metadata?.kind ?? undefined, + model: node.metadata?.model ?? undefined, + ratio: node.metadata?.ratio ?? undefined, + resolution: node.metadata?.resolution ?? undefined, + duration: node.metadata?.duration ?? undefined, + count: node.metadata?.count ?? undefined, + templateId: node.metadata?.templateId ?? undefined, + }, + }); + } + for (const conn of connections) { + const from = index.get(conn.fromNodeId); + const to = index.get(conn.toNodeId); + if (!from || !to) continue; + actions.push({ type: 'connect', args: { from: `@${from}`, to: `@${to}` } }); + } + return { id, title, actions }; +} + +/** + * The kind a shared template file says it is. + * + * A file carries what it is, so a file that is something else is refused + * instead of being read as a template. + */ +export const TEMPLATE_FILE_KIND = 'ai-node-canvas-template'; + +/** The shape of the file this version writes. */ +export const TEMPLATE_FILE_VERSION = 1; + +/** + * The steps a template may carry. + * + * A template is the skeleton of a chain, so it makes nodes and draws the + * lines between them. The settings of a node travel with the node that is + * made. Anything else — a run, a removal — is not a step of a chain. + */ +export const TEMPLATE_ACTION_TYPES = ['add_node', 'connect']; + +/** + * The document a template is shared as. + * + * A template travels as a file: the steps are in it, and nothing else is, so + * it works on another machine and does not come out as a canvas full of + * broken pictures. + */ +export function templateFile({ id = '', title = '', description = '', actions = [] } = {}) { + const steps = Array.isArray(actions) ? actions : []; + return { + kind: TEMPLATE_FILE_KIND, + version: TEMPLATE_FILE_VERSION, + id: String(id ?? ''), + title: String(title ?? ''), + description: String(description ?? ''), + steps: steps.length, + actions: steps, + }; +} + +/** + * Read a shared template file. + * + * The file is read by the one rule that puts a template on a canvas, so a + * template from another machine lands the same way as one made here. A file + * that is not a template is refused, and the reason names the rule it broke. + */ +export function readTemplateFile(text) { + let parsed; + try { + parsed = JSON.parse(String(text ?? '')); + } catch { + return { ok: false, error: '这个文件不是 JSON' }; + } + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + return { ok: false, error: '这个文件不是一份模板' }; + } + if (parsed.kind && parsed.kind !== TEMPLATE_FILE_KIND) { + return { ok: false, error: '这个文件不是节点画布的模板' }; + } + if (parsed.version && Number(parsed.version) > TEMPLATE_FILE_VERSION) { + return { ok: false, error: '这个模板的版本比这个画布新,请先更新画布' }; + } + const actions = parsed.actions; + if (!Array.isArray(actions) || !actions.length) { + return { ok: false, error: '这个模板里没有步骤' }; + } + for (const action of actions) { + if (!action || typeof action !== 'object' || !action.args || typeof action.args !== 'object') { + return { ok: false, error: '这个模板里有一步写坏了' }; + } + if (action.type !== 'add_node' && action.type !== 'connect') { + return { ok: false, error: `这个模板里有一步不是节点也不是连线:${action.type ?? '空'}` }; + } + } + return { + ok: true, + template: { + id: String(parsed.id ?? ''), + title: String(parsed.title ?? '').trim() || '导入的模板', + description: String(parsed.description ?? ''), + actions, + }, + }; +} + +/** + * Apply one action to a canvas document. + * + * The document is not changed in place: the caller decides when to write it. + * The result says what happened, so the caller can report it to whoever asked. + */ +export function applyAction(canvas, action, context = {}) { + const doc = canvas ?? { nodes: [], connections: [] }; + const nodes = [...(doc.nodes ?? [])]; + const connections = [...(doc.connections ?? [])]; + const ctx = { + sizes: {}, + modelsByType: {}, + defaultModelKey: null, + idFor: fallbackId, + ...context, + }; + const args = action?.args ?? {}; + + if (action?.type === 'add_node') { + const type = String(args.type ?? 'image'); + if (!CREATABLE_TYPES.includes(type)) { + return { + canvas: doc, + error: `没有这种节点:${type}(可以加的是:${CREATABLE_TYPES.join('、')})`, + }; + } + const id = String(args.id ?? ctx.idFor('n')); + const existing = nodes.find((node) => node.id === id); + if (existing) { + // The same action twice is not a second node. + if (existing.type !== type) { + return { canvas: doc, error: `这个画布上已经有 ${id} 了` }; + } + return { canvas: doc, info: { id, type, added: false } }; + } + if (nodes.length >= MAX_NODES) { + return { canvas: doc, error: `一张画布最多 ${MAX_NODES} 个节点` }; + } + const size = ctx.sizes[type] ?? { width: 300, height: 300, title: '节点' }; + const prompt = String(args.prompt ?? '').slice(0, MAX_PROMPT); + const mode = args.mode ? String(args.mode) : null; + // The template the prompt node expands its sentence with. The Agent names + // the same template the page would pick, so the two ways of making a + // prompt node end up the same. + const template = promptTemplateFor({ + mode: mode ?? '', + templateId: args.templateId, + templateText: args.templateText, + templates: ctx.promptTemplates, + }); + const node = { + id, + type, + title: String(args.title ?? size.title ?? '节点').slice(0, 80), + position: { + x: Math.round(Number(args.x ?? 80) || 0), + y: Math.round(Number(args.y ?? 80) || 0), + }, + width: Math.round(Number(args.width ?? size.width ?? 300)), + height: Math.round(Number(args.height ?? size.height ?? 300)), + metadata: { + templateId: template.templateId, + templateVars: {}, + segments: prompt ? [{ type: 'text', text: prompt }] : [], + prompt, + mentions: [], + // Only a generating node picks a model: an input node holds something + // and a result node shows something. + model: roleOf({ type }) === 'generate' ? pickModel(ctx, type) : null, + ratio: null, + resolution: null, + duration: null, + voice: null, + emotion: null, + sessionId: null, + // What the card holds: an input node holds what the user put in it, a + // result node holds what the generation produced, and a generating node + // holds nothing but its own settings. + kind: type === 'input' + ? (DATA_KINDS.includes(args.kind) ? args.kind : 'text') + : kindOf({ type }), + // The way the node makes its data: a text node writes text or expands + // a sentence into a prompt, an audio node speaks or makes music. + mode, + templateText: template.templateText, + output: { ...IDLE_OUTPUT }, + }, + }; + nodes.push(node); + return { + canvas: { ...doc, nodes }, + node, + info: { id, type, title: node.title, added: true }, + }; + } + + if (action?.type === 'apply_template') { + // A template is a list of actions, so putting one on the canvas is + // putting those actions on the canvas — by the same rules. + const applied = applyTemplate(doc, action.args?.template, ctx); + if (applied.error) return { canvas: doc, error: applied.error }; + return { + canvas: applied.canvas, + info: { nodes: applied.nodes, added: applied.nodes.length, id: action.args?.template?.id ?? '' }, + }; + } + + if (action?.type === 'connect') { + const from = nodes.find((node) => node.id === args.from); + const to = nodes.find((node) => node.id === args.to); + if (!from || !to) { + return { canvas: doc, error: '连线的一端不在这个画布上' }; + } + if (from.id === to.id) { + return { canvas: doc, error: '节点不能连到自己' }; + } + const kind = kindOf(from); + if (connections.some((conn) => conn.fromNodeId === from.id && conn.toNodeId === to.id)) { + // The same action twice is not a second line. + return { canvas: doc, info: { from: from.id, to: to.id, kind, added: false } }; + } + if (connections.length >= MAX_CONNECTIONS) { + return { canvas: doc, error: `一张画布最多 ${MAX_CONNECTIONS} 条连线` }; + } + // A line that closes a loop would let a run feed itself. + if (reaches(doc, to.id, from.id)) { + return { canvas: doc, error: '这样连会形成一个环' }; + } + // The ports are the mode's inputs, so a line whose kind the target does not + // take has nowhere to land. The target moves to a mode that takes it, or + // the line is refused with a reason. + const plan = planIncoming(doc, to, kind); + if (plan.error) { + return { canvas: doc, error: plan.error }; + } + if (plan.changed) { + const index = nodes.findIndex((node) => node.id === to.id); + nodes[index] = { ...to, metadata: { ...to.metadata, mode: plan.modeId } }; + } + // The port the line lands on. A mode that takes the same kind twice has a + // port for each, and the line says which one it is. + const target = nodes.find((node) => node.id === to.id) ?? to; + const port = planPort(doc, target, kind, String(args.role ?? '')); + if (!port.ok) { + return { canvas: doc, error: port.reason }; + } + const connection = { + id: String(args.id ?? ctx.idFor('c')), + fromNodeId: from.id, + toNodeId: to.id, + // The kind travels with the line, so the line is drawn on the port it + // belongs to instead of in the middle of the card. + kind, + role: port.role, + }; + connections.push(connection); + // A material the source's prompt names has to reach the node it now + // feeds, and the new target may be missing a material its own prompt + // names. The rule is the same one the page uses, so a line drawn by hand + // and a line the Agent asks for wire the material the same way. + const wired = syncMentions({ ...doc, nodes, connections }, from.id, ctx); + const wiredTarget = syncMentions(wired.canvas, to.id, ctx); + return { + canvas: wiredTarget.canvas, + connection, + info: { + id: connection.id, + from: from.id, + to: to.id, + kind, + added: true, + // The mode the target had to move into, so the caller can say so. + mode: plan.changed ? plan.modeId : '', + // The material the prompt names, wired in with the line. + addedNodes: wiredTarget.added.length + wired.added.length, + connectedLines: wiredTarget.connected.length + wired.connected.length, + }, + }; + } + + if (action?.type === 'remove_node') { + const id = String(args.id ?? ''); + if (!nodes.some((node) => node.id === id)) { + // The same action twice is not a second removal. + return { canvas: doc, info: { id, removed: false } }; + } + return { + canvas: { + ...doc, + nodes: nodes.filter((node) => node.id !== id), + connections: connections.filter( + (conn) => conn.fromNodeId !== id && conn.toNodeId !== id, + ), + }, + info: { id, removed: true }, + }; + } + + if (action?.type === 'update_node') { + const index = nodes.findIndex((node) => node.id === args.id); + if (index < 0) { + return { canvas: doc, error: '这个画布上没有这个节点' }; + } + const node = nodes[index]; + let next = { ...node, metadata: { ...node.metadata } }; + if (args.title !== undefined) next.title = String(args.title).slice(0, 80); + if (args.prompt !== undefined) { + const text = String(args.prompt).slice(0, MAX_PROMPT); + next.metadata.prompt = text; + // The prompt is shown as one text segment: the Agent writes plain text. + next.metadata.segments = text ? [{ type: 'text', text }] : []; + next.metadata.mentions = []; + } + if (args.model !== undefined) next.metadata.model = args.model ? String(args.model) : null; + if (args.ratio !== undefined) next.metadata.ratio = args.ratio ? String(args.ratio) : null; + if (args.resolution !== undefined) { + next.metadata.resolution = args.resolution ? String(args.resolution) : null; + } + if (args.duration !== undefined) { + const duration = Number(args.duration); + next.metadata.duration = Number.isFinite(duration) ? duration : null; + } + if (args.count !== undefined) { + const count = Number(args.count); + next.metadata.count = Number.isFinite(count) ? Math.max(1, Math.min(8, Math.round(count))) : 1; + } + if (args.voice !== undefined) next.metadata.voice = args.voice ? String(args.voice) : null; + if (args.emotion !== undefined) next.metadata.emotion = args.emotion ? String(args.emotion) : null; + if (args.templateId !== undefined || args.templateText !== undefined) { + // The template the prompt node expands its sentence with. Naming + // another one brings that one's text with it. + const template = promptTemplateFor({ + mode: next.metadata.mode ?? '', + templateId: args.templateId ?? next.metadata.templateId, + templateText: args.templateText, + templates: ctx.promptTemplates, + }); + next.metadata.templateId = template.templateId; + next.metadata.templateText = template.templateText; + } + let dropped = []; + let modeChanged = null; + if (args.mode !== undefined) { + // The mode is a choice the card offers; a mode the node does not have + // is not set on it — and it is said, because a change that quietly did + // nothing is a change the caller believes it made. + const modes = modesOf(next); + if (modes.some((mode) => mode.id === args.mode)) { + next.metadata.mode = String(args.mode); + // The model belongs to the mode: a music run uses the music model. + const models = ctx.modelsByType?.[runModeOf(next)] ?? []; + if (models.length && !models.includes(next.metadata.model)) { + next.metadata.model = models[0]; + } + // The ports are the mode's inputs, so a line the new mode does not take + // has nowhere to land: it goes, and the caller is told which ones. + dropped = linesDroppedByMode(doc, next, args.mode); + for (let i = connections.length - 1; i >= 0; i -= 1) { + if (dropped.includes(connections[i])) connections.splice(i, 1); + } + } else { + return { + canvas: doc, + error: `这个节点没有「${args.mode}」这个模式(${next.type} 节点有的是:${modes.map((mode) => mode.id).join('、')})`, + }; + } + } + if (args.model !== undefined) { + // The model decides the options: the values it does not offer are + // snapped onto ones it does, and a mode it cannot run is left for one it + // can. The same rule the page's model drop-down uses. + const result = applyModel(doc, next, args.model ? String(args.model) : null, ctx); + next = result.node; + dropped = dropped.concat(result.dropped); + for (let i = connections.length - 1; i >= 0; i -= 1) { + if (result.dropped.includes(connections[i])) connections.splice(i, 1); + } + if (result.mode.from !== result.mode.to) modeChanged = result.mode; + } + // The lines are handed to the ports of the mode the node is in now, so + // each one says which port it is on. A mode that takes the same kind + // twice — 图生视频·首尾帧 — has a port for each, and the line has to say + // which one it fills. + const kept = connections.filter((conn) => !dropped.includes(conn)); + const roles = stampPortRoles({ ...doc, connections: kept }, next); + for (const conn of kept) { + if (conn.toNodeId !== next.id) continue; + if (!roles.has(conn.id)) continue; + conn.role = roles.get(conn.id); + } + // The kind is not among them: it is what the node *is*, so it is fixed + // when the node is created. Changing it later would throw away whatever + // was already put in. + nodes[index] = next; + // The new mode may take a material the prompt names, and the node may + // not have it yet. + const wired = syncMentions({ ...doc, nodes, connections }, next.id, ctx); + return { + canvas: wired.canvas, + node: next, + info: { + id: next.id, + dropped: dropped.map((conn) => conn.id), + mode: modeChanged ? { from: modeChanged.from, to: modeChanged.to } : null, + // The material the prompt names, wired in with the change. + addedNodes: wired.added.length, + connectedLines: wired.connected.length, + }, + }; + } + + return { canvas: doc, error: `不认识的指令:${action?.type ?? ''}` }; +} + +/** + * The text a node holds, or produced. + * + * An input node's text is what was typed into it; anything else's is what it + * generated. + */ +export function textOfSource(source) { + if (!source) return ''; + if (roleOf(source) === 'input') { + return String(source.metadata?.output?.text || source.metadata?.prompt || '').trim(); + } + return String(source.metadata?.output?.text ?? '').trim(); +} + +/** + * The text a node gets from upstream. + * + * A line of text that lands on the node hands its text over: the node's text + * box shows that text and stops being the user's to type in. The text comes + * from the source that carries it, so a text input, a text result and a text + * node that has run all count. + * + * The newest line wins, because that is the one the user just drew. + */ +export function inheritedTextFor(canvas, node) { + const connections = canvas?.connections ?? []; + for (let i = connections.length - 1; i >= 0; i -= 1) { + const conn = connections[i]; + if (conn.toNodeId !== node?.id) continue; + const source = (canvas?.nodes ?? []).find((item) => item.id === conn.fromNodeId); + if (!source || kindOf(source) !== 'text') continue; + const text = textOfSource(source); + if (text) return { text, title: source.title, id: source.id }; + } + return null; +} + +/** + * The sentence a prompt node expands. + * + * The sentence is what the user wrote; the template is the shape the expanded + * prompt has to take. + */ +export function enhancePrompt(node, sentence) { + const template = String(node?.metadata?.templateText ?? '').trim(); + const line = String(sentence ?? node?.metadata?.prompt ?? '').trim(); + return [ + '你是一个提示词工程师。把下面这句话扩写成一条可以直接用于生成的提示词。', + template ? `必须遵守的模板:${template}` : '', + line ? `原始需求:${line}` : '', + '只输出扩写后的提示词本身,不要解释,不要加引号。', + ] + .filter(Boolean) + .join('\n'); +} + +/** + * What a run of this node takes with it. + * + * The chain above the node is the reference: every ancestor contributes its + * material, so an input three hops up still reaches the node. The rule is + * pure, so the page and the runtime send the same thing to the provider. + */ +export function runSpecFor(canvas, node, context = {}) { + if (!node) return { error: '这个画布上没有这个节点' }; + // The text that landed on the node's text box is the prompt now: the box + // shows it and cannot be typed in, so the run sends that text. + const inherited = inheritedTextFor(canvas, node); + const references = upstreamNodes(canvas, node.id) + .filter((source) => source.type !== 'group') + // The source that handed the text over is not a reference any more: its + // text is the prompt, and a text source carries no material to lose. + .filter((source) => source.id !== inherited?.id) + .map((source) => { + const role = roleOf(source); + return { + id: source.id, + title: source.title, + // What the data is, not what the card is called: a music node hands on + // audio, and a prompt node hands on text. + kind: kindOf(source), + role, + // The text of an input node is the prompt material; the text of + // anything else is what it generated. + text: role === 'input' + ? (source.metadata?.output?.text || source.metadata?.prompt || '') + : String(source.metadata?.output?.text ?? ''), + media: source.metadata?.output?.media ?? [], + }; + }); + // A material the prompt names is a reference too, even when it is not + // above the node: the `@` says "this prompt is about that picture", and + // the picture has to be there for the prompt to mean anything. + for (const material of mentionedMaterials(canvas, node)) { + if (references.some((item) => item.id === material.id)) continue; + references.push({ + id: material.id, + title: material.title, + kind: material.kind, + role: roleOf(material.node), + text: String(material.node?.metadata?.output?.text ?? ''), + media: material.media, + }); + } + // The prompt mode runs on the text model, and what it sends is the expanded + // prompt rather than the sentence the user typed. + const isPromptMode = modeOf(node) === 'prompt'; + // The node's own prompt stays in the node — it comes back when the upstream + // line is gone — but it is not sent this time: the text box shows the + // upstream text, so the run sends the upstream text. + const takenOver = Boolean(inherited); + return { + nodeId: node.id, + mode: runModeOf(node), + // The mode the node is in, not the way it runs: some modes need the + // runtime to read the material first (a video has to be understood before + // a prompt can be written from it). + nodeMode: modeOf(node), + // Which request the provider is asked for. The three video requests are + // different requests, not one request with different parameters, so the + // runtime assembles the body from this and not from the model alone. + path: pathOf(node), + // The pictures that are plugged straight into this node, and the port + // each one fills. A frame request wants the two lines on the card — the + // first frame and the last frame — and not everything above it. + frames: framePorts(canvas, node), + segments: isPromptMode || takenOver ? [] : (node.metadata?.segments ?? []), + prompt: isPromptMode + ? enhancePrompt(node, inherited?.text) + : (inherited?.text || node.metadata?.prompt || ''), + mentions: isPromptMode || takenOver ? [] : (node.metadata?.mentions ?? []), + params: { + ratio: node.metadata?.ratio ?? null, + resolution: node.metadata?.resolution ?? null, + duration: node.metadata?.duration ?? null, + voice: node.metadata?.voice ?? null, + emotion: node.metadata?.emotion ?? null, + }, + modelKey: node.metadata?.model ?? null, + sessionId: node.metadata?.sessionId ?? null, + // How many assets this run asks for. + count: batchCount(node.type, node.metadata?.count), + references, + // The whole chain is one run: the caller may fill in the steps above. + chain: context.chain !== false, + }; +} + +/** The result nodes a generating node produced. */ +function resultNodesOf(canvas, node) { + const ids = new Set(); + for (const conn of canvas?.connections ?? []) { + if (conn.fromNodeId !== node.id) continue; + const target = (canvas?.nodes ?? []).find((item) => item.id === conn.toNodeId); + if (target && target.type === 'result') ids.add(target.id); + } + return (canvas?.nodes ?? []).filter((item) => ids.has(item.id)); +} + +/** The generating node this node feeds, if there is one. */ +function downstreamGeneratingNode(canvas, node) { + for (const conn of canvas?.connections ?? []) { + if (conn.fromNodeId !== node.id) continue; + const target = (canvas?.nodes ?? []).find((item) => item.id === conn.toNodeId); + if (target && roleOf(target) === 'generate') return target; + } + return null; +} + +/** + * The result of a run becomes its own node. + * + * The node is placed to the right of the generating one, or in the middle when + * the generating one feeds another generating node — then the chain reads + * 生成 → 结果 → 生成, and the result is what the next node reads. + * + * A batch of results is one node per asset, in one row, so the versions can be + * compared next to each other. + */ +export function materializeResults(canvas, node, task, context = {}) { + if (!node) return { canvas, error: '没有可放结果的节点' }; + const ctx = { + sizes: {}, + idFor: fallbackId, + now: () => new Date().toISOString(), + ...context, + }; + const doc = canvas ?? { nodes: [], connections: [] }; + const nodes = [...(doc.nodes ?? [])]; + const connections = [...(doc.connections ?? [])]; + const media = task?.media ?? []; + + // The result is applied once: a second run of the same task does not add a + // second copy. + const already = nodes.find( + (item) => item.metadata?.sourceTaskId && item.metadata.sourceTaskId === task?.id, + ); + if (already) return { canvas: doc, results: [already], info: { added: false } }; + + // The prompt mode writes text, so its result is a text result. + const kind = media[0]?.kind ?? (node.type === 'text' ? 'text' : 'image'); + const size = ctx.sizes.result ?? { width: 300, height: 300 }; + + // A second run replaces the results of the first one instead of adding more: + // otherwise the downstream node would read the same step twice. + const previous = resultNodesOf(doc, node); + const carried = connections.filter((conn) => previous.some((item) => item.id === conn.fromNodeId)); + if (previous.length) { + const gone = new Set(previous.map((item) => item.id)); + for (let i = connections.length - 1; i >= 0; i -= 1) { + const conn = connections[i]; + if (gone.has(conn.fromNodeId) || gone.has(conn.toNodeId)) connections.splice(i, 1); + } + for (let i = nodes.length - 1; i >= 0; i -= 1) { + if (gone.has(nodes[i].id)) nodes.splice(i, 1); + } + } + + const downstream = downstreamGeneratingNode(doc, node); + const makeResult = (options) => ({ + id: options.id, + type: 'result', + title: options.title, + position: { x: options.x, y: options.y }, + width: size.width, + height: size.height, + metadata: { + templateId: 'custom', + templateVars: {}, + segments: [], + prompt: '', + mentions: [], + model: null, + ratio: null, + resolution: null, + duration: null, + voice: null, + emotion: null, + sessionId: null, + kind: options.kind, + templateText: '', + sourceNodeId: node.id, + sourceTaskId: task?.id ?? '', + output: { + ...IDLE_OUTPUT, + status: 'success', + text: options.text, + media: options.media, + updatedAt: ctx.now(), + }, + }, + }); + + if (media.length > 1) { + // The results sit in a row next to the node, and the distance between them + // is the card's own size: two result cards that are a few pixels apart + // cover each other. + const placed = planBatchPlacement({ + count: media.length, + originX: node.position.x + node.width + 60, + originY: node.position.y + 40, + cardWidth: size.width, + cardHeight: size.height, + perRow: media.length, + }); + const created = media.map((item, index) => { + const result = makeResult({ + id: ctx.idFor('n'), + title: `${node.title} · 结果 ${index + 1}`, + x: placed.cells[index].x, + y: placed.cells[index].y, + kind: item.kind || kind, + text: index === 0 ? task?.text || '' : '', + media: [item], + }); + nodes.push(result); + connections.push({ + id: ctx.idFor('c'), + fromNodeId: node.id, + toNodeId: result.id, + kind: result.metadata.kind, + }); + return result; + }); + const first = created[0]; + // The node that reads the results reads the first one, and it moves past + // the whole row so the row stays visible. + if (downstream) { + const last = created[created.length - 1]; + downstream.position.x = last.position.x + size.width + 60; + for (let i = connections.length - 1; i >= 0; i -= 1) { + const conn = connections[i]; + if (conn.fromNodeId === node.id && conn.toNodeId === downstream.id) connections.splice(i, 1); + } + connections.push({ + id: ctx.idFor('c'), + fromNodeId: first.id, + toNodeId: downstream.id, + kind: first.metadata.kind, + }); + } + // The links the replaced results carried move to the first new one. + const seen = new Set(); + for (const conn of carried) { + if (conn.toNodeId === downstream?.id) continue; + if (seen.has(conn.toNodeId)) continue; + seen.add(conn.toNodeId); + connections.push({ + id: ctx.idFor('c'), + fromNodeId: first.id, + toNodeId: conn.toNodeId, + kind: conn.kind ?? first.metadata.kind, + }); + } + // The cards never cover each other: a result that lands on another card + // pushes it aside. Two branches of one node are side by side, and the + // result of the first one would sit on top of the second one. + const apart = pushAside({ nodes, connections }, created.map((item) => item.id)); + return { + canvas: { ...doc, nodes: apart.nodes, connections: apart.connections }, + results: created, + info: { added: true, count: created.length }, + }; + } + + let x = node.position.x + node.width + 80; + if (downstream) { + const wanted = downstream.position.x - (size.width + 60); + if (wanted >= node.position.x + node.width + 24) { + x = wanted; + } else { + // No room between the two: the downstream node moves right so the chain + // reads left to right instead of the result sitting on top. + const shift = node.position.x + node.width + 24 + size.width + 60 - downstream.position.x; + if (shift > 0) downstream.position.x += shift; + x = downstream.position.x - size.width - 60; + } + } + + const result = makeResult({ + id: ctx.idFor('n'), + title: `${node.title} · 结果`, + x, + y: node.position.y + 40, + kind, + text: task?.text || '', + media, + }); + nodes.push(result); + connections.push({ + id: ctx.idFor('c'), + fromNodeId: node.id, + toNodeId: result.id, + kind: result.metadata.kind, + }); + // What the node fed now reads from the result, so the chain carries the data + // through the result node. + if (downstream) { + for (let i = connections.length - 1; i >= 0; i -= 1) { + const conn = connections[i]; + if (conn.fromNodeId === node.id && conn.toNodeId === downstream.id) connections.splice(i, 1); + } + connections.push({ + id: ctx.idFor('c'), + fromNodeId: result.id, + toNodeId: downstream.id, + kind: result.metadata.kind, + }); + } + // The links the replaced result carried move to the new one. + for (const conn of carried) { + if (conn.toNodeId === downstream?.id) continue; + connections.push({ + id: ctx.idFor('c'), + fromNodeId: result.id, + toNodeId: conn.toNodeId, + kind: conn.kind ?? result.metadata.kind, + }); + } + // The cards never cover each other: a result that lands on another card + // pushes it aside. Two branches of one node are side by side, and the + // result of the first one would sit on top of the second one. + const apart = pushAside({ nodes, connections }, [result.id]); + return { + canvas: { ...doc, nodes: apart.nodes, connections: apart.connections }, + results: [result], + info: { added: true, count: 1 }, + }; +} diff --git a/plugins/antianqi/ai-node-canvas/miniapp/client/app.js b/plugins/antianqi/ai-node-canvas/miniapp/client/app.js new file mode 100644 index 0000000..b520f9a --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/miniapp/client/app.js @@ -0,0 +1,5948 @@ +// AI Node Canvas - page runtime. +// +// One node is one asset slot: an image node holds an image, generating +// updates that node in place, and every media node carries its own prompt, +// parameters and generate button. A style node holds a preset whose text is +// prepended to the prompt of every downstream node. +// +// Rendering model: nodes are DOM elements with `data-node-id`, edges live in one +// SVG layer, the scene sits under a single CSS transform, and the prompt +// editor is an inline contentEditable so `@` references can be embedded as +// chips in the text. +import { detectMention, parseSegments, plainText, textIndexBefore } from './mention.mjs'; +import { bandSelection, BATCH_COUNTS, BATCH_MODES, cropRectAfterDrag, dragPositions, effectiveOption, fanOutParams, fitMediaBox, fitParamsToCaps, inputKindsFor, isControlTarget, kindOf, mentionCandidates as mentionCandidatesOf, modelSummary, NODE_GAP, placeOverlay, planBatchPlacement, planGridSplit, planPaste, portOfConnection, portState, pushAside, resolveEventTarget, roleOf, slotsOf as slotsOfForm, TILE_CARD, toolPrompt, upstreamNodes } from './tools.mjs'; +import { cardChrome, DATA_KINDS, hasTextBox, isTypingTarget, modeLabelOf, modesFor, NODE_FORMS, NODE_KINDS, modeOf, modeRecordOf, modesOf, portKinds, portKindsFor, portList, ratioLabel, ratioOptionsFor, redrawBlocker, runModeOf, stripInputs } from './forms.mjs'; +import { isBusyRefusal, runIsGone, waitForTheRun, withDeadline } from './run-watch.mjs'; +import { applyAction, applyModel, inheritedTextFor, lineKind, linesDroppedByMode, materializeResults, planIncoming, planPort, promptTemplateFor, readTemplateFile, runSpecFor, syncMentions, templateFile, templateFromCanvas } from './actions.mjs'; + +(() => { + 'use strict'; + + // The names and the menu order come from the node table, so a new node + // form is added in one place and the page cannot fall out of step with the + // runtime. + const KIND_LABEL = Object.fromEntries( + Object.entries(NODE_FORMS).map(([type, form]) => [type, form.label]), + ); + // The menu offers the kinds of node — a role and the data it moves — so + // what it lists is exactly what the board can show. A mode is a choice on + // the card, so it never gets a row of its own. + const KIND_ORDER = NODE_KINDS; + // The name of a way to run: it is a kind of node, not a raw mode key, so the + // capability list never shows "MUSIC" to the user. + const RUN_MODE_LABEL = { + image: '图片', + video: '视频', + audio: '音频', + music: '音频 · 音乐', + text: '文本', + }; + // The palette follows the system unless the user picks a side. + const THEME_ORDER = ['system', 'light', 'dark']; + const THEME_LABEL = { system: '跟随系统', light: '亮色', dark: '暗色' }; + const THEME_KEY = 'nxc-theme'; + const POLL_MS = 700; + const ACTION_POLL_MS = 700; + const AUTOSAVE_MS = 1100; + // A poll that never comes back would stop the whole loop, and the card would + // turn for ever on a run that is long gone. + const POLL_TIMEOUT_MS = 8000; + // A click that is in flight: the button under the pointer must not be + // replaced before the pointer lets go, or the click lands on nothing. The + // hold is only believed for a moment, because a pointer that was let go + // outside the window never sends its pointerup. + const POINTER_HOLD_MS = 5000; + const HISTORY_MAX = 50; + const HISTORY_DEBOUNCE_MS = 350; + const GRID_SIZE = 48; + const MIN_ZOOM = 0.05; + const MAX_ZOOM = 5; + const SNAP_SCREEN_PX = 56; + const DRAG_THRESHOLD = 3; + const MINI_W = 184; + const MINI_H = 124; + const CLIPBOARD_KIND = 'nxc-canvas-nodes'; + + // A group of cards the app placed together. + // + // The real size of a card is measured from the page: the preview box + // follows the media's own ratio and the tool bar wraps, so the height is + // not known before the card is on the board. A group is therefore placed + // again when a card of it is measured, and the cards stand by their real + // size instead of by a guess. + const layoutGroups = new Map(); + const layoutGroupOf = new Map(); + let layoutSeq = 0; + // The answer to the one question the page asks in itself. + let confirmResolve = null; + + const state = { + canvas: { + id: null, + title: '未命名画布', + nodes: [], + connections: [], + backgroundMode: 'lines', + viewport: { x: 80, y: 60, k: 1 }, + }, + selection: [], + editingTitleId: null, + canvases: [], + dropdown: null, + models: [], + defaultModelKey: null, + voices: [], + voiceError: null, + palette: null, + capabilities: {}, + capabilitiesByModel: {}, + media: { available: false, version: '', ffprobe: false, canUpload: false }, + tasks: new Map(), // taskId -> task record + nodeTasks: new Map(), // nodeId -> taskId + polling: new Set(), // taskIds whose poll is still on its way + pollTimer: null, + actionTimer: null, + saveTimer: null, + historyTimer: null, + snapNodeId: null, + draft: null, + mention: null, + panel: null, + history: { stack: [], index: -1, max: HISTORY_MAX }, + clip: null, + band: null, + pendingRender: false, + pendingPanel: false, + pointerAt: 0, + minimap: false, + theme: 'system', + editors: new Map(), + crop: null, // { nodeId, rect:{x,y,w,h} } in 0..1 image space + tool: null, // { nodeId, tool, params } + picker: null, // the node picker that opens on an empty drop + suppressClick: false, + }; + + const dom = {}; + let nodeSeq = 0; + let connSeq = 0; + + // ---------------------------------------------------------------- helpers + + const $ = (id) => document.getElementById(id); + + function el(tag, className, text) { + const node = document.createElement(tag); + if (className) node.className = className; + if (text !== undefined && text !== null) node.textContent = String(text); + return node; + } + + function uid(prefix) { + nodeSeq += 1; + connSeq += 1; + return `${prefix}_${Date.now().toString(36)}${Math.random().toString(36).slice(2, 7)}`; + } + + async function api(path, options = {}) { + const response = await fetch(path, { + headers: { 'content-type': 'application/json' }, + ...options, + body: options.body ? JSON.stringify(options.body) : undefined, + }); + const text = await response.text(); + let payload = null; + try { + payload = text ? JSON.parse(text) : null; + } catch { + payload = { error: 'bad_response', message: text.slice(0, 200) }; + } + if (!response.ok) { + const error = new Error(payload?.message || payload?.error || `HTTP ${response.status}`); + error.code = payload?.error || 'http_error'; + error.status = response.status; + if (response.status === 401) { + // The Host rotates the loopback credential when the runtime restarts, + // and that happens on every plugin update. The page still holds the + // old one, so every write is refused until the board is opened again. + // A bare "save failed" gives the user nothing to act on. + showBanner('页面会话已失效:插件更新后运行时会重启,请重新打开画布再继续。', 'error'); + } + throw error; + } + return payload; + } + + function setSaveState(next) { + dom.saveState.dataset.state = next; + dom.saveState.textContent = { + pending: '待保存', + saving: '保存中…', + saved: '已保存', + error: '保存失败', + }[next]; + } + + // ---------------------------------------------------------------- theme + + /** + * Apply the palette. + * + * `system` removes the attribute so the media query decides, and an + * explicit choice pins it. The page never forces one side. + */ + function applyTheme() { + const root = document.documentElement; + if (state.theme === 'system') root.removeAttribute('data-theme'); + else root.setAttribute('data-theme', state.theme); + try { + localStorage.setItem(THEME_KEY, state.theme); + } catch { + // A blocked storage must not stop the theme from applying. + } + } + + function loadTheme() { + let saved = 'system'; + try { + saved = localStorage.getItem(THEME_KEY) ?? 'system'; + } catch { + saved = 'system'; + } + state.theme = THEME_ORDER.includes(saved) ? saved : 'system'; + applyTheme(); + } + + function cycleTheme() { + const index = THEME_ORDER.indexOf(state.theme); + state.theme = THEME_ORDER[(index + 1) % THEME_ORDER.length]; + applyTheme(); + renderToolbar(); + } + + // ---------------------------------------------------------------- build check + + const PAGE_BUILD = window.__NXC_BUILD__ || ''; + + async function syncBuild() { + if (!PAGE_BUILD) return; + try { + const response = await fetch('/api/bootstrap'); + const payload = await response.json(); + const current = payload.clientBuild; + if (current && current !== PAGE_BUILD) { + const already = sessionStorage.getItem('nxc-reloaded-from'); + if (already !== PAGE_BUILD) { + sessionStorage.setItem('nxc-reloaded-from', PAGE_BUILD); + location.reload(); + return; + } + } + if (payload.pluginVersion || current) { + const version = payload.pluginVersion || window.__NXC_VERSION__ || ''; + dom.buildBadge.textContent = `${version ? `v${version} ` : ''}${current ?? PAGE_BUILD}`; + dom.buildBadge.title = `运行中的客户端构建指纹:${current ?? PAGE_BUILD}`; + } + } catch { + // The build check must never block the canvas from working. + } + } + + // ---------------------------------------------------------------- history + + /** + * The content of the canvas, for the history. + * + * The view is not part of it: panning the board is not a change to the + * document, and undoing a change should not move the camera. + */ + function snapshot() { + return JSON.stringify({ + nodes: state.canvas.nodes, + connections: state.canvas.connections, + }); + } + + function pushHistory() { + const h = state.history; + const s = snapshot(); + if (h.stack[h.index] === s) return; + h.stack = h.stack.slice(0, h.index + 1); + h.stack.push(s); + while (h.stack.length > h.max) h.stack.shift(); + h.index = h.stack.length - 1; + updateHistoryBtns(); + } + + function resetHistory() { + state.history = { stack: [snapshot()], index: 0, max: HISTORY_MAX }; + updateHistoryBtns(); + } + + function applySnapshot(s) { + let o; + try { + o = JSON.parse(s); + } catch { + return; + } + state.canvas.nodes = o.nodes ?? []; + state.canvas.connections = o.connections ?? []; + // The camera stays where it is: the history holds the document, not the + // place it is looked at from. + state.selection = state.selection.filter((id) => state.canvas.nodes.some((n) => n.id === id)); + render(); + markDirty(); + updateHistoryBtns(); + } + + function undo() { + const h = state.history; + if (h.index <= 0) { + showBanner('没有可撤销的操作'); + return; + } + h.index -= 1; + applySnapshot(h.stack[h.index]); + } + + function redo() { + const h = state.history; + if (h.index >= h.stack.length - 1) { + showBanner('没有可重做的操作'); + return; + } + h.index += 1; + applySnapshot(h.stack[h.index]); + } + + function updateHistoryBtns() { + const h = state.history; + dom.undoButton.disabled = h.index <= 0; + dom.redoButton.disabled = h.index >= h.stack.length - 1; + } + + // ---------------------------------------------------------------- graph + + function getNode(id) { + return state.canvas.nodes.find((node) => node.id === id) || null; + } + + // The whole chain above this node. The rule lives in `tools.mjs`, where it + // is pure, so the runtime reads the same chain as the page. + function upstreamOf(nodeId) { + return upstreamNodes(state.canvas, nodeId); + } + + function hasPath(fromId, toId) { + const stack = [fromId]; + const seen = new Set(); + while (stack.length) { + const current = stack.pop(); + if (current === toId) return true; + if (seen.has(current)) continue; + seen.add(current); + state.canvas.connections + .filter((conn) => conn.fromNodeId === current) + .forEach((conn) => stack.push(conn.toNodeId)); + } + return false; + } + + function nodeSize(type) { + return state.palette?.sizes?.[type] ?? { width: 300, height: 300 }; + } + + // The input kinds a generating node takes. It follows the model the node + // selected: a model that cannot take a video does not offer a video slot. + function inputKindsForNode(node) { + if (roleOf(node) !== 'generate') return []; + return inputKindsFor(effectiveCaps(node)); + } + + // The name of the mode the node is in. The mode is a choice on the card, + // so it names itself. + function nodeModeLabel(node) { + if (roleOf(node) !== 'generate') return ''; + return modeRecordOf(node)?.label ?? ''; + } + + // The generating nodes on their own. + function isGeneratingNode(node) { + return Boolean(node) && roleOf(node) === 'generate'; + } + + // The result node a generating node produced. + function resultNodeOf(node) { + if (!node) return null; + for (const conn of state.canvas.connections) { + if (conn.fromNodeId !== node.id) continue; + const target = getNode(conn.toNodeId); + if (target && target.type === 'result') return target; + } + return null; + } + + // Every result node a generating node produced. A batch of results is more + // than one, and a second run has to replace all of them. + function resultNodesOf(node) { + if (!node) return []; + const ids = new Set(); + for (const conn of state.canvas.connections) { + if (conn.fromNodeId !== node.id) continue; + const target = getNode(conn.toNodeId); + if (target && target.type === 'result') ids.add(target.id); + } + return state.canvas.nodes.filter((item) => ids.has(item.id)); + } + + // Whether the node already produced something. + // + // A result whose asset is gone does not count: the chain would reuse a dead + // reference instead of running that step again, and the step is cheap + // compared to a broken picture further down. + function hasResult(node) { + const result = resultNodeOf(node); + if (!result) return false; + return !result.metadata?.output?.missing; + } + + // The text an upstream line hands over is read in `actions.mjs`, where the + // page and the runtime share the rule: the card shows the upstream text and + // the run sends that same text. + + // A video node is generated by the connector with a dedicated video model, + // while every other mode is driven by an Agent turn that runs on a text + // model. The capability table carries both lists; the model dropdown has to + // show the one that belongs to the node, otherwise a video node offers + // MiniMax-M3 and never the video models it can actually use. + function modeModelOptions(node) { + // The models belong to the way the node makes its data: an audio node in + // the music mode runs the music model. + const models = state.capabilities[runModeOf(node)]?.models ?? []; + if (models.length === 0) return []; + return models.map((model) => ({ value: model.id, label: model.label, hint: model.note })); + } + + function agentModelOptions() { + return state.models.map((model) => ({ + value: model.key, + label: model.miniMax ? `${model.label} ★` : model.label, + })); + } + + // The limits of the model a node actually selected: Hailuo offers other + // resolutions and durations than H3, and the parameter row has to follow. + // + // The runtime resolves them once and ships them per model, so the rule + // that a model only offers a resolution for some durations is written + // down in one place only. + function effectiveCaps(node) { + const runMode = runModeOf(node); + const base = state.capabilities[runMode] ?? {}; + const models = base.models ?? []; + if (models.length === 0) return base; + const resolved = state.capabilitiesByModel?.[runMode]?.[node.metadata.model]; + if (!resolved) return base; + const caps = { ...resolved }; + // A model may only offer a resolution for particular durations; the + // parameter row has to show what the connector would actually accept. + const allowed = resolved.resolutionByDuration?.[String(node.metadata.duration)]; + if (allowed) caps.resolutions = allowed; + return caps; + } + + function createNode(type, position, options = {}) { + const size = nodeSize(type); + // Only a generating node picks a model; an input node holds something + // and a result node shows something. The model belongs to the way the node + // makes its data: a music run uses the music model. + const generates = !['input', 'result', 'group'].includes(type); + const mode = options.mode ?? modeOf({ type, metadata: {} }); + const runMode = runModeOf({ type, metadata: { mode } }); + const modeModels = state.capabilities[runMode]?.models ?? []; + const node = { + id: uid('n'), + type, + title: options.title ?? KIND_LABEL[type] ?? '节点', + position: { + x: Math.round(position?.x ?? 120), + y: Math.round(position?.y ?? 120), + }, + width: size.width, + height: size.height, + metadata: { + templateVars: {}, + segments: [], + prompt: '', + mentions: [], + // The mode's own model list wins; the Agent catalog is the fallback. + model: generates ? (modeModels.length ? modeModels[0].id : state.defaultModelKey) : null, + // The way the node makes its data: a text node writes text or expands + // a sentence into a prompt, an audio node speaks or makes music. + mode: mode || null, + ratio: null, + resolution: null, + duration: null, + voice: null, + emotion: null, + sessionId: null, + // What the card holds: an input node holds what the user put in it, a + // result node holds what the generation produced. A generating node + // holds nothing but its own settings. + kind: options.kind ?? (type === 'result' ? 'image' : 'text'), + // The template the prompt card expands the sentence with. The page + // and the Agent ask the same rule, so a prompt node is the same + // either way. + ...promptTemplateFor({ + mode: mode || '', + templateId: options.templateId, + templateText: options.templateText, + templates: state.palette?.promptTemplates ?? [], + }), + output: { status: 'idle', text: '', media: [], error: '', posterUrl: '', durationMs: 0, width: 0, height: 0, missing: false, updatedAt: '' }, + }, + }; + if (options.output) { + node.metadata.output = { ...node.metadata.output, ...options.output }; + } + state.canvas.nodes.push(node); + return node; + } + + function removeNode(id) { + state.canvas.nodes = state.canvas.nodes.filter((node) => node.id !== id); + state.canvas.connections = state.canvas.connections.filter( + (conn) => conn.fromNodeId !== id && conn.toNodeId !== id, + ); + state.selection = state.selection.filter((item) => item !== id); + } + + function copyNode(node) { + if (!node) return; + const clone = { + ...node, + id: uid('n'), + title: `${node.title} 副本`, + position: { x: node.position.x + 40, y: node.position.y + 40 }, + metadata: { + ...node.metadata, + sessionId: null, + output: { status: 'idle', text: '', media: [], error: '', posterUrl: '', durationMs: 0, width: 0, height: 0, missing: false, updatedAt: '' }, + }, + }; + state.canvas.nodes.push(clone); + state.selection = [clone.id]; + markDirty(); + render(); + } + + function removeConnection(id) { + state.canvas.connections = state.canvas.connections.filter((conn) => conn.id !== id); + markDirty(); + render(); + } + + /** + * The kind a line carries, read live from its source. + * + * The line may have been drawn before the source changed what it holds, so + * the source is the truth. + */ + function kindOfConnection(conn) { + return lineKind(state.canvas, conn); + } + + /** + * Draw a line between two nodes. + * + * The target's mode says what it takes. A line of a kind the mode does not + * take moves the target to the mode that takes it — that is how a picture + * turns an image node into a 图生图 node. When no mode takes it, the line is + * refused and the reason is given. + * + * The rule itself lives in `actions.mjs`, where the page and the runtime + * share it, so a line drawn by hand and a line the Agent asks for land in the + * same shape. + * + * `quiet` leaves the redraw to the caller. A batch that draws many lines at + * once — a local tool that cuts a video into thirty pieces — would otherwise + * rebuild the whole board thirty times. + */ + function connect(fromId, toId, kind = '', options = {}) { + if (!fromId || !toId || fromId === toId) return { ok: false, reason: '这条线连不到自己身上。' }; + if (state.canvas.connections.some((c) => c.fromNodeId === fromId && c.toNodeId === toId)) { + return { ok: false, reason: '这两个节点已经连上了。' }; + } + if (hasPath(toId, fromId)) return { ok: false, reason: '这样连会形成一个环。' }; + const target = getNode(toId); + let role = options.role ?? ''; + if (target && kind) { + const plan = planIncoming(state.canvas, target, kind); + if (plan.error) return { ok: false, reason: plan.reason }; + if (plan.changed) { + target.metadata.mode = plan.modeId; + // The card changed shape, so the user is told what happened to it. + showBanner(`「${target.title}」切到「${modeLabelOf(target, plan.modeId)}」,这样才接得住这条线。`); + } + // The port the line lands on. A mode that takes the same kind twice has + // a port for each, and the role says which one this line is. + const port = planPort(state.canvas, target, kind, role); + if (!port.ok) return { ok: false, reason: port.reason }; + role = port.role; + } + // The kind travels with the line, so the line can be drawn on the port it + // belongs to instead of in the middle of the card. + state.canvas.connections.push({ id: uid('c'), fromNodeId: fromId, toNodeId: toId, kind, role }); + // A material the source's prompt names has to reach the node it now + // feeds, and the new target may be missing a material its own prompt + // names. The rule lives in `actions.mjs`, so the Agent and the page wire + // it the same way. + applyMentions(fromId); + applyMentions(toId); + if (!options.quiet) { + markDirty(); + render(); + } + return { ok: true, reason: '' }; + } + + /** + * Wire the material a prompt names into the nodes that need it. + * + * A mention says the prompt is about that material, so the node that + * generates from the prompt has to receive it as an input of its own — + * otherwise the prompt names a picture the request never sends. The rule + * lives in `actions.mjs`, so the Agent and the page wire it the same way. + */ + function applyMentions(nodeId) { + if (!nodeId) return false; + const synced = syncMentions(state.canvas, nodeId, actionContext()); + if (synced.added.length === 0 && synced.connected.length === 0) return false; + state.canvas.nodes = synced.canvas.nodes; + state.canvas.connections = synced.canvas.connections; + markDirty(); + return true; + } + + // ---------------------------------------------------------------- viewport + + function applyViewport() { + const { x, y, k } = state.canvas.viewport; + dom.world.style.transform = `translate(${x}px, ${y}px) scale(${k})`; + const size = GRID_SIZE * k; + dom.grid.style.backgroundSize = `${size}px ${size}px`; + dom.grid.style.backgroundPosition = `${x % size}px ${y % size}px`; + const dot = k < 0.12 ? 0.9 : 1.2; + const line = k < 0.12 ? 0.16 : 0.1; + dom.grid.style.backgroundImage = + state.canvas.backgroundMode === 'dots' + ? `radial-gradient(circle, rgba(148,163,184,${line + 0.14}) ${dot}px, transparent ${dot + 0.4}px)` + : `linear-gradient(rgba(148,163,184,${line}) 1px, transparent 1px), linear-gradient(90deg, rgba(148,163,184,${line}) 1px, transparent 1px)`; + dom.zoom.textContent = `${Math.round(k * 100)}%`; + renderMinimap(); + } + + function screenToWorld(clientX, clientY) { + const rect = dom.viewport.getBoundingClientRect(); + const { x, y, k } = state.canvas.viewport; + return { x: (clientX - rect.left - x) / k, y: (clientY - rect.top - y) / k }; + } + + /** + * The world position of a port. + * + * A generating node can have several input ports, so a line has to start + * and end on the port itself and not on the middle of the node. + * + * A text line is the exception: a node with a text box takes its text in the + * box, so the line ends on the box and not on a port. + */ + function portPointOf(node, side, kind = '', role = '') { + if (side === 'in' && kind === 'text' && hasTextBox(node)) { + const box = dom.nodes.querySelector(`[data-node-id="${node.id}"] [data-text-box]`); + if (box) { + const rect = box.getBoundingClientRect(); + return screenToWorld(rect.left, rect.top + rect.height / 2); + } + } + // A mode that takes the same kind twice has two ports of it, and the role + // is what tells them apart. + const selector = role + ? `[data-node-id="${node.id}"] [data-port="${side}"][data-kind="${kind}"][data-role="${role}"]` + : kind + ? `[data-node-id="${node.id}"] [data-port="${side}"][data-kind="${kind}"]` + : null; + const byKind = selector ? dom.nodes.querySelector(selector) : null; + const el = byKind ?? dom.nodes.querySelector(`[data-node-id="${node.id}"] [data-port="${side}"]`); + if (!el) { + return { + x: side === 'in' ? node.position.x : node.position.x + node.width, + y: node.position.y + node.height / 2, + }; + } + const rect = el.getBoundingClientRect(); + return screenToWorld(side === 'in' ? rect.left : rect.right, rect.top + rect.height / 2); + } + + // The same, for the port element the pointer went down on. + function portPointOfEl(el, side) { + const rect = el.getBoundingClientRect(); + return screenToWorld(side === 'in' ? rect.left : rect.right, rect.top + rect.height / 2); + } + + function centerOfViewport() { + const rect = dom.viewport.getBoundingClientRect(); + return screenToWorld(rect.left + rect.width / 2, rect.top + rect.height / 2); + } + + // ---------------------------------------------------------------- minimap + + function minimapBox() { + const nodes = state.canvas.nodes; + let minX = Infinity; + let minY = Infinity; + let maxX = -Infinity; + let maxY = -Infinity; + for (const node of nodes) { + minX = Math.min(minX, node.position.x); + minY = Math.min(minY, node.position.y); + maxX = Math.max(maxX, node.position.x + node.width); + maxY = Math.max(maxY, node.position.y + node.height); + } + const vp = state.canvas.viewport; + const rect = dom.viewport.getBoundingClientRect(); + const vw = rect.width / vp.k; + const vh = rect.height / vp.k; + minX = Math.min(minX, -vp.x / vp.k); + minY = Math.min(minY, -vp.y / vp.k); + maxX = Math.max(maxX, -vp.x / vp.k + vw); + maxY = Math.max(maxY, -vp.y / vp.k + vh); + if (!Number.isFinite(minX)) return null; + minX -= 60; + minY -= 60; + maxX += 60; + maxY += 60; + const k = Math.min(MINI_W / Math.max(1, maxX - minX), MINI_H / Math.max(1, maxY - minY)); + return { + minX, + minY, + k, + ox: (MINI_W - (maxX - minX) * k) / 2, + oy: (MINI_H - (maxY - minY) * k) / 2, + }; + } + + function renderMinimap() { + if (!state.minimap || !dom.minimap) return; + const ctx = dom.minimap.getContext('2d'); + ctx.clearRect(0, 0, MINI_W, MINI_H); + const box = minimapBox(); + if (!box) return; + for (const node of state.canvas.nodes) { + const w = node.width * box.k; + const h = node.height * box.k; + const x = (node.position.x - box.minX) * box.k + box.ox; + const y = (node.position.y - box.minY) * box.k + box.oy; + ctx.fillStyle = + node.type === 'group' + ? 'rgba(167,139,250,.20)' + : state.selection.includes(node.id) + ? 'rgba(34,211,238,.60)' + : 'rgba(120,140,170,.45)'; + ctx.fillRect(x, y, Math.max(2, w), Math.max(2, h)); + } + const vp = state.canvas.viewport; + const rect = dom.viewport.getBoundingClientRect(); + const vx = (-vp.x / vp.k - box.minX) * box.k + box.ox; + const vy = (-vp.y / vp.k - box.minY) * box.k + box.oy; + ctx.strokeStyle = 'rgba(34,211,238,.95)'; + ctx.lineWidth = 1; + ctx.strokeRect(vx, vy, Math.max(6, (rect.width / vp.k) * box.k), Math.max(6, (rect.height / vp.k) * box.k)); + } + + function minimapJump(event) { + const box = minimapBox(); + if (!box) return; + const rect = dom.minimap.getBoundingClientRect(); + const worldX = (event.clientX - rect.left - box.ox) / box.k + box.minX; + const worldY = (event.clientY - rect.top - box.oy) / box.k + box.minY; + const vpRect = dom.viewport.getBoundingClientRect(); + state.canvas.viewport = { + x: vpRect.width / 2 - worldX * state.canvas.viewport.k, + y: vpRect.height / 2 - worldY * state.canvas.viewport.k, + k: state.canvas.viewport.k, + }; + applyViewport(); + markDirty({ history: false }); + } + + // ---------------------------------------------------------------- composer + + function plainTextOf(segments) { + return plainText(segments); + } + + function buildChip(segment) { + const source = getNode(segment.nodeId); + const media = source?.metadata?.output?.media?.[0]; + const orphan = !media; + const chip = el( + 'span', + `nxc-mention-chip nxc-mention-chip--${segment.kind}${orphan ? ' nxc-mention-chip--orphan' : ''}`, + ); + chip.contentEditable = 'false'; + chip.dataset.mentionNodeId = segment.nodeId; + chip.dataset.mentionKind = segment.kind; + if (segment.kind === 'image' && media?.url && !orphan) { + const img = el('img', 'nxc-mention-chip__thumb'); + img.src = media.url; + img.alt = ''; + img.loading = 'lazy'; + img.decoding = 'async'; + img.draggable = false; + chip.appendChild(img); + } else if (segment.kind === 'video' && media?.url && !orphan) { + const video = el('video', 'nxc-mention-chip__thumb'); + video.src = media.url; + video.muted = true; + video.playsInline = true; + video.preload = 'metadata'; + chip.appendChild(video); + } else { + const thumb = el('span', 'nxc-mention-chip__thumb nxc-mention-chip__thumb--audio'); + thumb.textContent = segment.kind === 'audio' ? '♪' : '·'; + chip.appendChild(thumb); + } + chip.appendChild(el('span', null, source?.title ?? '(素材已删除)')); + if (orphan) chip.title = '素材已不在画布里,生成时会忽略它'; + return chip; + } + + function renderSegments(editor, segments) { + editor.textContent = ''; + for (const segment of segments) { + if (segment.type === 'text') editor.appendChild(document.createTextNode(segment.text)); + else editor.appendChild(buildChip(segment)); + } + } + + function caretTextIndex(editor) { + const selection = window.getSelection(); + if (!selection || selection.rangeCount === 0) return plainTextOf(parseSegments(editor)).length; + const range = selection.getRangeAt(0); + return textIndexBefore(editor, range.startContainer, range.startOffset); + } + + function insertMention(node, editor, candidate) { + const segments = parseSegments(editor); + const caret = caretTextIndex(editor); + let consumed = 0; + let inserted = false; + const next = []; + for (const segment of segments) { + const length = segment.type === 'text' ? segment.text.length : 1; + if (!inserted && caret <= consumed + length) { + if (segment.type === 'text') { + const offset = Math.max(0, Math.min(segment.text.length, caret - consumed)); + if (segment.text.slice(0, offset)) next.push({ type: 'text', text: segment.text.slice(0, offset) }); + next.push({ type: 'mention', nodeId: candidate.id, kind: candidate.kind }); + if (segment.text.slice(offset)) next.push({ type: 'text', text: segment.text.slice(offset) }); + } else { + next.push(segment); + next.push({ type: 'mention', nodeId: candidate.id, kind: candidate.kind }); + } + inserted = true; + continue; + } + next.push(segment); + consumed += length; + } + if (!inserted) next.push({ type: 'mention', nodeId: candidate.id, kind: candidate.kind }); + renderSegments(editor, next); + syncSegments(node, editor); + // The prompt now names that material, so the nodes that generate from + // this prompt have to receive it. The board is not redrawn here: the + // editor has the focus, and a redraw would take the caret away. The + // next redraw shows it. + applyMentions(node.id); + // The editor sits in the transformed layer, so the focus must not scroll + // the canvas to the editor's layout box. + editor.focus({ preventScroll: true }); + } + + function syncSegments(node, editor) { + const segments = parseSegments(editor); + node.metadata.segments = segments; + node.metadata.prompt = plainTextOf(segments); + node.metadata.mentions = segments + .filter((s) => s.type === 'mention') + .map((s) => ({ nodeId: s.nodeId, kind: s.kind })); + // An input node holds what goes in, so its text is its content: the + // downstream node reads it as the prompt material. + if (node.type === 'input') { + node.metadata.output.text = node.metadata.prompt; + node.metadata.output.status = 'success'; + node.metadata.output.updatedAt = new Date().toISOString(); + } + } + + /** + * The sources a mention can point at. + * + * The whole canvas is in the list, not only the chain above the node: a + * prompt may be about a picture that sits somewhere else, and naming it is + * what wires it in. The rule lives in `tools.mjs`, so the page and the + * runtime read the same one. + */ + function mentionCandidates(node) { + return mentionCandidatesOf(state.canvas, node); + } + + function closeMentionMenu() { + state.mention = null; + dom.mentionMenu.hidden = true; + } + + function openMentionMenu(node, editor, query) { + const items = mentionCandidates(node).filter((item) => + query ? item.title.toLowerCase().includes(query.toLowerCase()) : true, + ); + if (items.length === 0) { + closeMentionMenu(); + return; + } + state.mention = { nodeId: node.id, editor, items, activeIndex: 0 }; + const menu = dom.mentionMenu; + menu.textContent = ''; + items.forEach((item, index) => { + const button = el('button', `nxc-mention-menu__item${index === 0 ? ' is-active' : ''}`); + button.type = 'button'; + if (item.url) { + const img = el('img', 'nxc-mention-menu__thumb'); + img.src = item.url; + img.alt = ''; + button.appendChild(img); + } else { + button.appendChild(el('span', 'nxc-mention-menu__thumb', item.kind === 'audio' ? '♪' : '♪')); + } + button.appendChild(el('span', 'nxc-mention-menu__label', item.title)); + // The whole board is in the list, so the menu says which of them are + // already above this node — those are the ones a prompt is usually + // about, and the ones that need no new line. + if (item.upstream) { + button.appendChild(el('span', 'nxc-mention-menu__hint', '上游')); + } + button.addEventListener('mousedown', (event) => { + event.preventDefault(); + event.stopPropagation(); + }); + button.addEventListener('click', (event) => { + event.preventDefault(); + event.stopPropagation(); + insertMention(node, editor, item); + closeMentionMenu(); + }); + menu.appendChild(button); + }); + const rect = editor.getBoundingClientRect(); + // The mention menu is placed before it is shown, so its size is an + // estimate. The estimate still goes through the shared rule, so the + // menu cannot land above the top of the window. + const place = placeOverlay( + { left: rect.left, top: rect.bottom + 6 }, + { width: 258, height: 220 }, + { width: window.innerWidth, height: window.innerHeight }, + ); + menu.style.left = `${place.left}px`; + menu.style.top = `${place.top}px`; + menu.hidden = false; + } + + function maybeOpenMentionMenu(node, editor) { + const query = detectMention(parseSegments(editor), caretTextIndex(editor)); + if (query === null) closeMentionMenu(); + else openMentionMenu(node, editor, query); + } + + /** + * The prompt an upstream text line handed over. + * + * It is shown read-only: the text comes from upstream, so there is nothing + * to type here any more. The node keeps its own prompt — it comes back when + * the line is gone — so the user is told that it is still there. + */ + function renderInheritedPrompt(node, inherited) { + const box = el('div', 'nxc-inherited'); + box.dataset.canvasNoZoom = 'true'; + // The text box is where a text line lands, so the line ends here. + box.dataset.textBox = '1'; + box.appendChild(el('div', 'nxc-inherited__label', `来自上游:${inherited.title}`)); + box.appendChild(el('div', 'nxc-inherited__text', inherited.text)); + if (String(node.metadata?.prompt ?? '').trim()) { + box.appendChild(el('div', 'nxc-inherited__note', '你自己写的提示词还留着:断开这条线就能继续改。')); + } + return box; + } + + /** + * The upload control of an input node. + * + * The file is sent to the Node service, which keeps it under its own data + * directory and serves it back. The provider cannot reach that address, so + * the runtime uploads the file to the provider when it is used as a + * reference. + * + * Only a node that holds material has one: a text node holds text, which + * is typed or filled from a template, so it has nothing to pick. + */ + function renderUpload(node) { + const wrap = el('div', 'nxc-upload'); + wrap.dataset.canvasNoZoom = 'true'; + + const picker = el('input', 'nxc-upload__file'); + picker.type = 'file'; + const kind = node.metadata.kind ?? 'image'; + const label = { image: '图片', video: '视频', audio: '音频' }[kind] ?? '素材'; + // The input card only takes what its kind is for. + picker.accept = kind === 'image' ? 'image/*' : kind === 'video' ? 'video/*' : 'audio/*'; + picker.multiple = kind === 'image'; + picker.title = `把本机的${label}放进这个输入节点`; + picker.setAttribute('aria-label', `选择本地${label}`); + picker.addEventListener('change', () => { + const files = [...(picker.files ?? [])]; + picker.value = ''; + if (files.length) void uploadFiles(node, files); + }); + wrap.appendChild(picker); + + const media = node.metadata.output?.media ?? []; + if (media.length) { + const list = el('div', 'nxc-upload__list'); + for (const item of media) { + const row = el('div', 'nxc-upload__item'); + row.appendChild(el('span', 'nxc-upload__name', item.name || item.url)); + const remove = el('button', 'nxc-upload__remove', '×'); + remove.type = 'button'; + remove.title = '移除这个素材'; + remove.addEventListener('click', (event) => { + event.stopPropagation(); + node.metadata.output.media = media.filter((entry) => entry !== item); + node.metadata.output.updatedAt = new Date().toISOString(); + markDirty(); + render(); + }); + row.appendChild(remove); + list.appendChild(row); + } + wrap.appendChild(list); + } + return wrap; + } + + async function uploadFiles(node, files) { + showBanner(`正在上传 ${files.length} 个文件…`); + let failed = 0; + for (const file of files) { + try { + // The node's kind travels with the request: an input node holds one + // kind of material, so the wrong file is refused by the runtime. + const kind = node.metadata.kind ?? 'image'; + const response = await fetch( + `/api/uploads?name=${encodeURIComponent(file.name)}&kind=${encodeURIComponent(kind)}`, + { + method: 'POST', + headers: { 'content-type': file.type || 'application/octet-stream' }, + body: file, + }, + ); + const payload = await response.json().catch(() => null); + if (!response.ok) throw new Error(payload?.message || `HTTP ${response.status}`); + const output = node.metadata.output; + output.media = [ + ...(output.media ?? []), + { url: payload.url, kind: payload.kind, name: payload.originalName, local: true }, + ]; + output.status = 'success'; + output.updatedAt = new Date().toISOString(); + } catch (error) { + failed += 1; + showBanner(`「${file.name}」上传失败:${error?.message ?? error}`, 'error'); + } + } + if (!failed) showBanner(`${files.length} 个文件已上传`); + markDirty(); + render(); + } + + /** + * The template the prompt node expands the sentence with: a preset from the + * palette, or one the user writes. + */ + function renderPromptTemplate(node) { + const wrap = el('div', 'nxc-template'); + wrap.dataset.canvasNoZoom = 'true'; + const templates = state.palette?.promptTemplates ?? []; + const current = templates.find((item) => item.id === node.metadata.templateId) ?? templates[0]; + + const dropdown = createDropdown({ + options: templates.map((item) => ({ value: item.id, label: item.label })), + value: node.metadata.templateId || current?.id, + onChange: (value) => { + const preset = templates.find((item) => item.id === value); + node.metadata.templateId = value; + node.metadata.templateText = preset?.text ?? ''; + markDirty(); + render(); + }, + }); + wrap.appendChild(dropdown); + + const field = el('textarea', 'nxc-template__text'); + field.value = node.metadata.templateText ?? ''; + field.placeholder = '写自己的模板要求;留空则只按原句扩写'; + field.rows = 2; + field.setAttribute('aria-label', '提示词模板'); + field.addEventListener('input', () => { + node.metadata.templateText = field.value; + const preset = templates.find((item) => item.id === node.metadata.templateId); + if (!preset || preset.text !== field.value) node.metadata.templateId = 'custom'; + markDirty(); + }); + wrap.appendChild(field); + return wrap; + } + + // The prompt the prompt node sends to the text model is built in + // `actions.mjs`, where the page and the runtime share it. + + function createComposer(node) { + const wrap = el('div', 'nxc-composer'); + // The text box is where a text line lands: the line ends here, not on a + // port on the edge of the card. + wrap.dataset.textBox = '1'; + const mentionWrap = el('div', 'nxc-composer__mention-wrap'); + const placeholder = el('div', 'nxc-composer__placeholder', placeholderFor(node)); + const editor = el('div', 'nxc-composer__editor'); + editor.contentEditable = 'true'; + editor.setAttribute('role', 'textbox'); + editor.setAttribute('aria-multiline', 'true'); + // A placeholder is a hint, not a name: the editor needs one of its own. + editor.setAttribute('aria-label', `${KIND_LABEL[node.type] ?? node.type}节点的提示词`); + editor.dataset.canvasNoZoom = 'true'; + + renderSegments(editor, node.metadata.segments ?? []); + state.editors.set(node.id, editor); + let composing = false; + + editor.addEventListener('compositionstart', () => { + composing = true; + }); + editor.addEventListener('compositionend', () => { + composing = false; + syncSegments(node, editor); + maybeOpenMentionMenu(node, editor); + markDirty(); + }); + editor.addEventListener('input', () => { + if (composing) return; + syncSegments(node, editor); + maybeOpenMentionMenu(node, editor); + markDirty(); + }); + editor.addEventListener('keydown', (event) => { + event.stopPropagation(); + if (event.nativeEvent?.isComposing) return; + if (state.mention) { + if (event.key === 'ArrowDown') { + event.preventDefault(); + state.mention.activeIndex = (state.mention.activeIndex + 1) % state.mention.items.length; + paintMentionActive(); + return; + } + if (event.key === 'ArrowUp') { + event.preventDefault(); + state.mention.activeIndex = + (state.mention.activeIndex - 1 + state.mention.items.length) % state.mention.items.length; + paintMentionActive(); + return; + } + if (event.key === 'Enter' || event.key === 'Tab') { + event.preventDefault(); + const item = state.mention.items[state.mention.activeIndex]; + if (item) insertMention(node, editor, item); + closeMentionMenu(); + return; + } + if (event.key === 'Escape') { + event.preventDefault(); + closeMentionMenu(); + return; + } + } + if (event.key === 'Escape') editor.blur(); + }); + editor.addEventListener('blur', () => { + window.setTimeout(() => { + if (!state.mention) closeMentionMenu(); + flushRender(); + pushHistory(); + }, 120); + }); + + mentionWrap.appendChild(placeholder); + mentionWrap.appendChild(editor); + wrap.appendChild(mentionWrap); + wrap.appendChild(el('div', 'nxc-composer__hint', '输入 @ 可引用上游节点的产物')); + return wrap; + } + + function paintMentionActive() { + const items = dom.mentionMenu.querySelectorAll('.nxc-mention-menu__item'); + items.forEach((item, index) => { + item.classList.toggle('is-active', index === state.mention?.activeIndex); + }); + } + + function placeholderFor(node) { + const hasUpstream = upstreamOf(node.id).length > 0; + const mode = modeOf(node); + if (mode === 'prompt') return '一句话说明你想要什么,例如:一只猫在窗边打盹'; + if (mode === 'video2text') return '要反推的重点,例如:只写运镜和光线'; + if (mode === 'music') return '例如:轻快电子口播 BGM,120 BPM,无人声'; + if (node.type === 'input') return '写要送进去的内容'; + if (node.type === 'text') return '描述要生成的文本内容'; + if (node.type === 'video') return hasUpstream ? '描述要生成的视频内容,输入 @ 引用上游素材' : '描述要生成的视频内容'; + if (node.type === 'audio') return '描述要生成的音频内容'; + return hasUpstream ? '描述要生成的图片内容,输入 @ 引用上游接入的图…' : '描述要生成的图片内容'; + } + + // ---------------------------------------------------------------- node render + + /** The kind of a node and the mode it is in, as the header shows it. */ + function nodeKindLabel(node) { + const mode = nodeModeLabel(node); + const kind = KIND_LABEL[node.type] ?? node.type; + if (node.type === 'input') { + return `输入 · ${state.palette?.inputKindLabels?.[node.metadata.kind] ?? node.metadata.kind}`; + } + return mode ? `${kind} · ${mode}` : kind; + } + + /** + * The accessible name of a node. + * + * It names the node and says whether it is selected, so the selection is + * carried by words as well and not by the border colour alone. + */ + /** + * Whether a node takes part in the current selection. + * + * While something is selected, the rest of the board steps back, so the + * selection and the lines around it are what the eye lands on. + */ + function isRelatedToSelection(nodeId) { + if (state.selection.length === 0) return true; + if (state.selection.includes(nodeId)) return true; + return state.canvas.connections.some((conn) => ( + (state.selection.includes(conn.fromNodeId) && conn.toNodeId === nodeId) + || (state.selection.includes(conn.toNodeId) && conn.fromNodeId === nodeId) + )); + } + + /** + * The slots of a node: what may be plugged into it, and what it hands on. + * + * The rule itself lives in `forms.mjs`, where the node table is, so the page + * and the runtime declare the same slots. + */ + function slotsOf(node) { + const slots = slotsOfForm(node); + // The strip reads the same rule the ports are drawn from: the text goes + // into the card's text box, and a kind the model cannot take is not an + // input either. + return { inputs: stripInputs(node, effectiveCaps(node)), outputs: slots.outputs }; + } + + /** + * The slot strip under the header: what may come in, and what goes out. + * + * An open slot stays hollow and says it is waiting, so what is still + * missing can be read off the card without touching anything. + */ + function renderSlotStrip(node, slots) { + const strip = el('div', 'nxc-slots'); + if (slots.inputs.length > 0) { + const inRow = el('div', 'nxc-slots__row'); + inRow.appendChild(el('span', 'nxc-slots__label', '入')); + for (const slot of slots.inputs) { + inRow.appendChild(renderSlot(slot, portState(state.canvas, node, slot))); + } + strip.appendChild(inRow); + } + if (slots.outputs.length > 0) { + const outRow = el('div', 'nxc-slots__row nxc-slots__row--out'); + for (const slot of slots.outputs) { + // An output is what the node hands on, so it is never waiting. + outRow.appendChild(renderSlot(slot, 'out')); + } + outRow.appendChild(el('span', 'nxc-slots__label', '出')); + strip.appendChild(outRow); + } + return strip; + } + + /** + * One slot: a dot in the colour of the data, and the name of that data. + * + * The state is carried by the shape of the dot, by the word next to it and + * by the tooltip, so it does not depend on colour alone. + * + * A port that has a role says which one it is: 首帧 and 尾帧 are both + * pictures, and the user has to be able to tell them apart. + */ + function renderSlot(slot, slotState = 'filled') { + const kind = slot?.kind ?? slot; + const open = slotState === 'open'; + const item = el('span', `nxc-slot${open ? ' is-open' : ''}${slotState === 'out' ? ' is-out' : ''}`); + item.dataset.kind = kind; + item.dataset.state = slotState; + const label = KIND_LABEL[kind] ?? kind; + const name = slot?.label ? `${slot.label} ${label}` : label; + item.title = { + filled: `${name} · 已接入`, + ready: `${name} · 已就绪`, + open: `${name} · 待接入`, + out: `${name} · 产出`, + }[slotState] ?? name; + item.appendChild(el('i', 'nxc-slot__dot')); + item.appendChild(el('span', 'nxc-slot__name', name)); + if (open) item.appendChild(el('span', 'nxc-slot__wait', '待接入')); + return item; + } + + function nodeAccessibleName(node) { + const kind = nodeKindLabel(node); + // A node that still carries its default title would say its kind twice. + const title = node.title && node.title !== kind ? node.title : ''; + const name = [kind, title].filter(Boolean).join(' '); + return state.selection.includes(node.id) ? `${name}(已选)` : name; + } + + function renderNode(node) { + // What the model this node selected can take. The ports and the modes + // follow it, so the card never offers something the run would refuse. + const caps = effectiveCaps(node); + const wrap = el('div', 'nxc-node'); + wrap.dataset.nodeId = node.id; + wrap.classList.add(`nxc-node--${node.type}`); + const selected = state.selection.includes(node.id); + if (selected) wrap.classList.add('nxc-node--selected'); + // A node is a group of controls, so it gets a name of its own. + wrap.setAttribute('role', 'group'); + if (node.id === state.snapNodeId) wrap.classList.add('nxc-node--connect-snap'); + const status = node.metadata.output?.status; + if (status === 'loading') wrap.classList.add('nxc-node--loading'); + if (status === 'error') wrap.classList.add('nxc-node--error'); + // The hue of the card, the ports and the line: what the node carries. + wrap.dataset.kind = kindOf(node); + // The state, as a colour next to the words that say it. + wrap.dataset.state = { + loading: 'running', + success: 'done', + error: 'error', + interrupted: 'interrupted', + }[status] ?? 'idle'; + // While something is selected, the rest of the board steps back. + if (!isRelatedToSelection(node.id)) wrap.classList.add('is-dim'); + wrap.style.left = `${node.position.x}px`; + wrap.style.top = `${node.position.y}px`; + wrap.style.width = `${node.width}px`; + wrap.style.height = `${node.height}px`; + + const head = el('div', 'nxc-node__head'); + // A dot that says how the node is doing, beside the words that say it. + head.appendChild(el('i', 'nxc-node__state')); + // The kind, and the mode the node is in: 文生图 becomes 图生图 when a + // picture is connected to it. An input card names what it takes. + const kindLabel = nodeKindLabel(node); + const kind = el('span', 'nxc-node__kind', kindLabel); + kind.dataset.kind = node.type; + head.appendChild(kind); + + if (state.editingTitleId === node.id) { + const input = el('input', 'nxc-node__title-input'); + input.value = node.title; + input.addEventListener('blur', () => { + state.editingTitleId = null; + node.title = input.value.trim() || node.title; + markDirty(); + window.setTimeout(() => scheduleRender(), 0); + }); + input.addEventListener('keydown', (event) => { + event.stopPropagation(); + if (event.key === 'Enter') input.blur(); + if (event.key === 'Escape') { + input.value = node.title; + input.blur(); + } + }); + head.appendChild(input); + window.setTimeout(() => { + // The card is a transformed layer: focusing the field must not scroll + // the canvas to the field's layout box. + input.focus({ preventScroll: true }); + input.select(); + }, 0); + } else { + const title = el('span', 'nxc-node__title', node.title); + title.addEventListener('dblclick', (event) => { + event.stopPropagation(); + state.editingTitleId = node.id; + render(); + }); + head.appendChild(title); + } + + // The selection is not only a border colour: the node says so in words, + // and its name carries the same state for a reader that cannot see it. + const chip = el('span', 'nxc-node__sel', '已选'); + chip.hidden = !selected; + head.appendChild(chip); + wrap.setAttribute('aria-label', nodeAccessibleName(node)); + + const task = state.tasks.get(state.nodeTasks.get(node.id)); + if (task && (task.status === 'running' || task.status === 'waiting-question')) { + const badge = el('span', 'nxc-node__status'); + badge.dataset.tone = 'running'; + badge.appendChild(el('i', 'nxc-spin')); + badge.appendChild(document.createTextNode(`${Math.round((task.elapsedMs ?? 0) / 1000)}s`)); + head.appendChild(badge); + } else if (status && status !== 'idle') { + const badge = el('span', 'nxc-node__status'); + badge.dataset.tone = status === 'success' ? 'success' : status === 'error' ? 'error' : 'running'; + badge.textContent = { success: '已完成', error: '失败', interrupted: '已中断' }[status] ?? status; + // A batch that lost an asset says so on the card: the user asked for + // four and three came out, and the card has to say it. + const asked = Number(node.metadata.count ?? 1); + const made = Array.isArray(task?.media) ? task.media.length : 0; + if (status === 'success' && asked > 1 && made > 0 && made < asked) { + badge.textContent = `已完成 ${made}/${asked}`; + badge.dataset.tone = 'error'; + badge.title = `这一批里少了 ${asked - made} 张`; + } + head.appendChild(badge); + } + wrap.appendChild(head); + + // The slots: what may come in and what goes out, read off the card. + const slots = slotsOf(node); + if (slots.inputs.length > 0 || slots.outputs.length > 0) { + wrap.appendChild(renderSlotStrip(node, slots)); + } + + const body = el('div', 'nxc-node__body'); + const role = roleOf(node); + if (role === 'group') { + body.appendChild(el('div', 'nxc-note', '把相关的节点框进来')); + } else if (role === 'input') { + // The input card holds what goes in: one kind of material. What kind it + // is, the header says — there is no switch for it, because the kind is + // fixed when the node is created. + const kind = node.metadata.kind ?? 'text'; + if (kind === 'text') { + // A text node holds text: the template fills it and the composer + // holds it, so there is nothing else on the card. + body.appendChild(renderTemplatePicker(node)); + body.appendChild(createComposer(node)); + } else { + // A node that holds material shows what it holds and lets the user + // pick more. It has no text box: the text that goes with the material + // is another node's job — a text input above it, or the prompt of the + // node it feeds. + body.appendChild(renderOutput(node)); + body.appendChild(renderUpload(node)); + } + } else if (role === 'result') { + // The result card only shows what was produced. + body.appendChild(renderOutput(node)); + } else { + // The generating card has the prompt, the model and the button, and + // no result of its own: the result arrives as its own node. + // A text line that landed on this node hands its text over, so the box + // shows that text instead of the user's own prompt. + const inherited = inheritedTextFor(state.canvas, node); + if (inherited) { + body.appendChild(renderInheritedPrompt(node, inherited)); + } else { + body.appendChild(createComposer(node)); + } + // The prompt mode of a text node brings the template picker with it. + if (modeOf(node) === 'prompt') body.appendChild(renderPromptTemplate(node)); + body.appendChild(renderParams(node)); + } + if (node.metadata.output?.notice) { + body.appendChild(el('div', 'nxc-node__notice', node.metadata.output.notice)); + } + if (node.metadata.output?.error) { + body.appendChild(el('div', 'nxc-node__error', node.metadata.output.error)); + } + wrap.appendChild(body); + bindNodeGestures(node, wrap); + + if (role === 'result') { + // The footer is the toolbar. A card with no tools has no footer, so no + // empty strip with a border is left under the material. + const bar = renderToolBar(node); + if (bar) { + const footer = el('div', 'nxc-node__footer'); + footer.appendChild(bar); + wrap.appendChild(footer); + } + } + + if (node.type !== 'group') { + // One port per input kind the node can take, so the picture of what + // this node needs is on the card itself. A node with a text box takes + // its text in the text box, so the text has no port: the line lands on + // the box instead. And a kind the model cannot take has no port either: + // the card must not offer something the run would refuse. + const ports = portList(node, caps); + const labels = state.palette?.inputKindLabels ?? {}; + ports.forEach((port, index) => { + const portIn = el('button', 'nxc-node__port nxc-node__port--in'); + portIn.type = 'button'; + portIn.dataset.port = 'in'; + portIn.dataset.kind = port.kind; + // A port that has a role says which one it is, so the line that + // lands on it says the same thing. + if (port.role) portIn.dataset.role = port.role; + // The port and the slot strip read the same state, so the card never + // says two different things about the same input. The name is not + // `state` on purpose: that would shadow the app state inside its own + // initializer and the card would fail to render. + const slotState = portState(state.canvas, node, port); + if (slotState !== 'open') portIn.classList.add('is-connected'); + const portLabel = port.label ? `${port.label} ` : ''; + portIn.title = `接入${portLabel}${labels[port.kind] ?? port.kind}(${ + slotState === 'filled' ? '已接入' : slotState === 'ready' ? '已就绪' : '未接入' + })`; + portIn.setAttribute('aria-label', portIn.title); + // The ports are spread around the middle of the left edge. + portIn.style.top = `${50 + (index - (ports.length - 1) / 2) * 13}%`; + wrap.appendChild(portIn); + }); + + const portOut = el('button', 'nxc-node__port nxc-node__port--out'); + portOut.type = 'button'; + portOut.dataset.port = 'out'; + portOut.dataset.kind = kindOf(node); + portOut.title = `输出${KIND_LABEL[kindOf(node)] ?? kindOf(node)}`; + portOut.setAttribute('aria-label', portOut.title); + wrap.appendChild(portOut); + } + + const resize = el('span', 'nxc-node__resize'); + resize.dataset.resize = 'se'; + wrap.appendChild(resize); + + if (status === 'loading') { + const ns = 'http://www.w3.org/2000/svg'; + const comet = document.createElementNS(ns, 'svg'); + comet.setAttribute('class', 'nxc-node__comet'); + comet.setAttribute('width', String(node.width + 2)); + comet.setAttribute('height', String(node.height + 2)); + comet.setAttribute('aria-hidden', 'true'); + const circle = document.createElementNS(ns, 'circle'); + circle.setAttribute('cx', String(node.width / 2)); + circle.setAttribute('cy', String(node.height / 2)); + circle.setAttribute('r', String(Math.max(node.width, node.height) / 2 + 4)); + comet.appendChild(circle); + wrap.appendChild(comet); + } + + return wrap; + } + + // The preview keeps the media's own ratio and the card is sized around it. + // The box itself is worked out by `fitMediaBox`, which is a pure function + // and is covered by the test suite. + const MEDIA_CHROME = 20; // the body padding and the preview border + + function applyMediaBox(mediaWrap, media, box) { + mediaWrap.classList.add('nxc-node__media-wrap--fitted'); + mediaWrap.style.width = `${box.width}px`; + mediaWrap.style.height = `${box.height}px`; + if (media && media.tagName !== 'AUDIO') { + media.style.width = '100%'; + media.style.height = '100%'; + } + } + + /** + * Size the node around the media it just produced. + * + * The preview box takes the media's own ratio, the card grows to fit it, + * and a card the user never resized also follows the media's width. + */ + function fitNodeToMedia(node, card, mediaWrap, media, naturalWidth, naturalHeight) { + if (!naturalWidth || !naturalHeight) return; + const output = node.metadata.output ?? (node.metadata.output = {}); + const signature = `${naturalWidth}x${naturalHeight}`; + if (output.fittedSignature === signature && output.fittedSize) return; + + const box = fitMediaBox(naturalWidth / naturalHeight, Math.max(160, node.width - MEDIA_CHROME)); + applyMediaBox(mediaWrap, media, box); + output.fittedSize = box; + output.fittedSignature = signature; + + // A card that was never resized by hand follows the media; one that was + // keeps the width the user chose. + if (!node.metadata.userSized) { + const width = Math.max(240, Math.min(420, box.width + MEDIA_CHROME)); + if (Math.abs(width - node.width) > 1) { + node.width = width; + card.style.width = `${width}px`; + } + } + + // The card grows with the preview: its natural height is measured with + // the media box already in place. + const previous = card.style.height; + card.style.height = 'auto'; + const natural = card.offsetHeight; + card.style.height = previous; + const height = Math.max(200, Math.min(2000, Math.round(natural))); + if (Math.abs(height - node.height) > 1) { + node.height = height; + card.style.height = `${height}px`; + markDirty(); + renderConnections(); + } + // The card is measured now, so the cards that were placed with it are + // placed again by their real size. + const groupId = layoutGroupOf.get(node.id); + if (groupId) relayoutGroup(groupId); + } + + /** + * The name a file is saved under. + * + * The title of the node it belongs to, so the file on the desktop says + * what it is, and the extension of the material, so the machine knows + * what it is. + */ + function fileNameFor(node, item) { + const ext = (/\.[A-Za-z0-9]{2,5}$/.exec(String(item?.url ?? '')) ?? [])[0] ?? ''; + const base = String(node?.title ?? '素材').replace(/[\\/:*?"<>|]/g, '').trim().slice(0, 60); + return `${base || '素材'}${ext}`; + } + + /** + * Hand the material of a node to the user's machine. + * + * The runtime keeps the file so the canvas can point at it, but that folder + * is not somewhere a person looks for a picture. The runtime copies it into + * the Downloads folder under the name the node carries, and says where it + * landed. + */ + async function saveMedia(node, item) { + const url = String(item?.url ?? ''); + if (!url) return false; + const file = url.split('/').pop() ?? ''; + if (!file) return false; + try { + const result = await api(`/api/library/${encodeURIComponent(file)}/save`, { + method: 'POST', + body: { name: fileNameFor(node, item) }, + }); + showBanner(`已保存到 ${result.path}`); + return true; + } catch (error) { + showBanner(`保存失败:${error?.message ?? error}`, 'error'); + return false; + } + } + + /** The row under a material: the way to get the file out of the app. */ + function renderMediaActions(node, item) { + const row = el('div', 'nxc-node__media-actions'); + const save = el('button', 'nxc-btn nxc-btn--small', '保存'); + save.type = 'button'; + save.dataset.canvasNoZoom = 'true'; + save.title = '把这个文件存到电脑'; + save.addEventListener('click', (event) => { + event.stopPropagation(); + saveMedia(node, item); + }); + row.appendChild(save); + return row; + } + + function renderOutput(node) { + const output = node.metadata.output ?? {}; + const box = el('div', 'nxc-node__output'); + if (output.media?.length) { + box.classList.add('nxc-node__output--media'); + const item = output.media[0]; + const mediaWrap = el('div', 'nxc-node__media-wrap'); + mediaWrap.dataset.canvasNoZoom = 'true'; + if (item.kind === 'video') { + const video = el('video', 'nxc-node__media'); + video.src = item.url; + if (output.posterUrl) video.poster = output.posterUrl; + video.muted = true; + video.playsInline = true; + video.preload = 'metadata'; + video.addEventListener('loadedmetadata', () => { + // The file is back: the card is whole again. + if (output.missing) { + output.missing = false; + markDirty(); + render(); + } + output.durationMs = Math.round((video.duration || 0) * 1000); + output.width = video.videoWidth || 0; + output.height = video.videoHeight || 0; + const card = mediaWrap.closest('.nxc-node'); + if (card) fitNodeToMedia(node, card, mediaWrap, video, video.videoWidth, video.videoHeight); + }); + video.addEventListener('seeked', () => { + try { + const canvas = document.createElement('canvas'); + canvas.width = video.videoWidth; + canvas.height = video.videoHeight; + const ctx = canvas.getContext('2d'); + if (!ctx) return; + ctx.drawImage(video, 0, 0); + output.posterUrl = canvas.toDataURL('image/jpeg', 0.82); + markDirty(); + } catch { + // A cross-origin video cannot be sampled; that is expected. + } + }); + video.addEventListener('error', () => { + if (output.missing) return; + output.missing = true; + markDirty(); + render(); + }); + mediaWrap.appendChild(video); + const play = el('button', 'nxc-node__video-play'); + play.type = 'button'; + play.dataset.canvasNoZoom = 'true'; + play.textContent = '▶'; + play.addEventListener('click', (event) => { + event.stopPropagation(); + if (video.paused) { + video.muted = false; + video.play().catch(() => {}); + } else { + video.pause(); + video.muted = true; + } + }); + mediaWrap.appendChild(play); + if (output.durationMs) { + mediaWrap.appendChild(el('span', 'nxc-node__video-duration', formatDuration(output.durationMs))); + } + } else if (item.kind === 'audio') { + const audio = el('audio', 'nxc-node__media nxc-node__media--audio'); + audio.src = item.url; + audio.controls = true; + audio.preload = 'metadata'; + mediaWrap.appendChild(audio); + } else { + const img = el('img', 'nxc-node__media'); + img.src = item.url; + img.alt = ''; + img.draggable = false; + img.addEventListener('load', () => { + // The file is back: the card is whole again, and the chain may use + // this step instead of running it once more. The card is drawn + // again, because the tools of a card whose material was lost are + // blocked until the material is there. + if (output.missing) { + output.missing = false; + markDirty(); + render(); + } + output.width = img.naturalWidth || 0; + output.height = img.naturalHeight || 0; + const card = mediaWrap.closest('.nxc-node'); + if (card) fitNodeToMedia(node, card, mediaWrap, img, img.naturalWidth, img.naturalHeight); + }); + img.addEventListener('error', () => { + // The asset is gone: say so instead of leaving a broken image. + // The guard matters: a re-render builds a new image element, which + // would fail again and start the loop over. + if (output.missing) return; + output.missing = true; + markDirty(); + render(); + }); + mediaWrap.appendChild(img); + if (state.crop?.nodeId === node.id) mediaWrap.appendChild(renderCropOverlay(node, mediaWrap)); + } + // A media that was already measured keeps the box it was given, so the + // card does not jump back to a fixed size on every render. + if (output.fittedSize && item.kind !== 'audio') { + applyMediaBox(mediaWrap, mediaWrap.querySelector('.nxc-node__media'), output.fittedSize); + } + // The file this card points at is gone. The card says so in words + // instead of showing a broken picture, and the picture stays in the + // page so the card heals by itself when the file is back. + if (output.missing) { + mediaWrap.classList.add('is-missing'); + mediaWrap.appendChild(el('div', 'nxc-media-lost', '素材已不在了。重新生成上游那一步,或者从资产库把它插回来。')); + } + box.appendChild(mediaWrap); + // The way out of the app: the file is on this machine, and the user + // may want it on theirs. + box.appendChild(renderMediaActions(node, item)); + } else if (output.missing) { + box.appendChild(el('div', 'nxc-media-lost', '素材已不在了')); + } else if (output.text) { + box.classList.add('nxc-node__output--text'); + box.appendChild(el('div', 'nxc-node__text-body', output.text)); + } else if (output.status !== 'loading' && output.status !== 'error') { + box.appendChild(el('div', 'nxc-note', emptyHintFor(node))); + } + return box; + } + + function emptyHintFor(node) { + const mode = modeOf(node); + // An input node does not generate anything: it holds what goes in. Only + // the nodes that hold material show this; a text node says it in its + // own box. + if (roleOf(node) === 'input') return '这里放你要送进去的素材:上传一个文件。'; + if (mode === 'prompt') return '写好要扩写的句子,点「生成」。'; + if (mode === 'music') return '写一段音乐描述点「生成」。'; + if (node.type === 'text') return '写好提示词点「生成」。'; + if (node.type === 'video') return '写好提示词点「生成」出视频。'; + if (node.type === 'audio') return '写好要念的文案点「生成」。'; + return '还没出图。写好提示词点「生成」。'; + } + + function formatDuration(ms) { + const total = Math.round(ms / 1000); + return `${Math.floor(total / 60)}:${String(total % 60).padStart(2, '0')}`; + } + + /** + * Snap the parameters onto what the model offers. + * + * The runtime snaps them again when it runs, so a value that is not allowed + * would be changed behind the user's back. The page does it at the moment a + * parameter changes, so what is shown is what will be generated. + */ + function fitNodeParams(node, caps) { + Object.assign(node.metadata, fitParamsToCaps(node.metadata, caps)); + } + + /** + * Put the node in another mode. + * + * The ports are the mode's inputs, so a line the new mode does not take has + * nowhere to land: it goes, and the user is told. The rule lives in + * `actions.mjs`, so the runtime drops the same lines. + */ + function renderParams(node) { + const caps = effectiveCaps(node); + const row = el('div', 'nxc-row'); + + // The mode is a choice on the card: it says what the node takes, so the + // ports on the left follow it. And a mode the model cannot run is not + // offered at all: only MiniMax-H3 can continue a video. + const modes = modesFor(node, caps); + if (modes.length) { + const field = paramSelect('模式', modeOf(node), modes.map((mode) => ({ value: mode.id, label: mode.label })), (value) => { + // The same action the Agent sends, so the page and the runtime drop + // the same lines. + const applied = applyAction( + state.canvas, + { type: 'update_node', args: { id: node.id, mode: value } }, + actionContext(), + ); + if (applied.error) { + showBanner(`切不了这个模式:${applied.error}`, 'error'); + return; + } + state.canvas.nodes = applied.canvas.nodes; + state.canvas.connections = applied.canvas.connections; + // The prompt mode starts with the first preset, so it is useful + // before anything is typed. + if (value === 'prompt') { + const at = state.canvas.nodes.findIndex((item) => item.id === node.id); + if (at >= 0) { + const template = promptTemplateFor({ + mode: 'prompt', + templates: state.palette?.promptTemplates ?? [], + }); + state.canvas.nodes[at].metadata.templateId = template.templateId; + state.canvas.nodes[at].metadata.templateText = template.templateText; + } + } + if (applied.info?.dropped?.length) { + showBanner(`切到「${nodeModeLabel(node)}」后,去掉了 ${applied.info.dropped.length} 条连线(这个模式不接)。`, 'error'); + } + // The new mode may take a material the prompt names, and the node may + // not have it yet. + applyMentions(node.id); + markDirty(); + render(); + // The card was rebuilt, so the control that was used is gone and the + // focus fell back to the page — where the next Tab would land on + // another node. Put the focus back on this node's mode. + focusModeControl(node); + }); + // The control is found again by this name after the card is rebuilt. + field.dataset.field = 'mode'; + row.appendChild(field); + } + + // The ratio follows the request the mode makes. The two keyframe requests + // follow the frame, so there is nothing to choose there. + const ratioOptions = ratioOptionsFor(node, caps); + if (ratioOptions.length) { + row.appendChild( + // The ratio says which way the frame lies, so the menu names it: the + // value stays the ratio the provider takes. + paramSelect('比例', node.metadata.ratio, ratioOptions.map((ratio) => ({ + value: ratio, + label: ratioLabel(ratio), + })), (value) => { + node.metadata.ratio = value; + }), + ); + } + if (caps.resolutions?.length) { + row.appendChild( + paramSelect('清晰度', node.metadata.resolution, caps.resolutions, (value) => { + node.metadata.resolution = value; + }), + ); + } + if (caps.supportsDuration && caps.durations?.length) { + row.appendChild( + paramSelect('时长', node.metadata.duration == null ? null : String(node.metadata.duration), caps.durations.map(String), (value) => { + node.metadata.duration = Number(value); + // The resolution depends on the duration, so it is snapped too. + fitNodeParams(node, effectiveCaps(node)); + // The row shows what will be generated right away, not on the next + // render tick. + render(); + }), + ); + } + if (caps.supportsVoice) { + // The voice catalogue is live data the Node service reads from the + // connector; until it arrives the row says so instead of offering a + // made-up list. + const voiceOptions = state.voices.length + ? state.voices + : [{ value: '', label: state.voiceError ?? '音色清单读取中…' }]; + row.appendChild( + paramSelect('音色', node.metadata.voice ?? 'male-qn-qingse', voiceOptions, (value) => { + node.metadata.voice = value; + }), + ); + row.appendChild( + paramSelect( + '情绪', + node.metadata.emotion ?? '', + [{ value: '', label: '默认' }, ...(caps.emotions ?? [])], + (value) => { + node.metadata.emotion = value || null; + }, + ), + ); + } + row.appendChild(modelSelect(node)); + + // How many assets one run produces. It sits right next to the button that + // starts the run, because that is where the choice matters. + if (BATCH_MODES.includes(runModeOf(node))) { + row.appendChild( + paramSelect('数量', String(node.metadata.count ?? 1), BATCH_COUNTS, (value) => { + node.metadata.count = Number(value); + }), + ); + } + + const running = isRunning(node.id); + const generate = el('button', 'nxc-btn nxc-btn--primary', running ? '生成中…' : '生成'); + generate.type = 'button'; + generate.dataset.canvasNoZoom = 'true'; + generate.disabled = running || !state.agentAvailable; + generate.addEventListener('click', (event) => { + event.stopPropagation(); + void startRun(node); + }); + row.appendChild(generate); + + // A node that ran through an Agent turn keeps its session, so the + // conversation is still reachable after the run is over. + if (node.metadata.sessionId) { + if (pageAgent()?.open) { + const chat = el('button', 'nxc-btn', '对话'); + chat.type = 'button'; + chat.dataset.canvasNoZoom = 'true'; + chat.title = '在对话里打开这次生成所在的会话'; + chat.addEventListener('click', (event) => { + event.stopPropagation(); + void openInChat(node.metadata.sessionId); + }); + row.appendChild(chat); + } + + // A new turn in the same conversation carries the old one along. This + // drops the root session, so the next run starts a fresh one. It is + // disabled while a run is in flight: the run in flight would put its + // own session back on the node. + const fresh = el('button', 'nxc-btn', '新会话'); + fresh.type = 'button'; + fresh.dataset.canvasNoZoom = 'true'; + fresh.disabled = running; + fresh.title = running + ? '当前这次生成还在跑,跑完才能换新会话' + : '下次生成从新会话开始,不再接着这次对话'; + fresh.addEventListener('click', (event) => { + event.stopPropagation(); + node.metadata.sessionId = null; + markDirty(); + render(); + showBanner('下次生成会开一个新会话'); + }); + row.appendChild(fresh); + } + + const wrap = el('div', 'nxc-node__params'); + wrap.appendChild(row); + + // The video node has no sound switches: the mode says what the request + // carries (a reference with an audio or without one), and the model's own + // note says whether the picture comes with a track. A switch the request + // ignores is a switch the user cannot trust. + + const refs = upstreamOf(node.id); + if (refs.length > 0) { + const counts = { image: 0, video: 0, audio: 0 }; + for (const ref of refs) { + const kind = ref.metadata.output.media?.[0]?.kind; + if (kind && counts[kind] !== undefined) counts[kind] += 1; + } + const text = [ + counts.image ? `${counts.image} 图` : '', + counts.video ? `${counts.video} 视频` : '', + counts.audio ? `${counts.audio} 音频` : '', + ] + .filter(Boolean) + .join(' · '); + wrap.appendChild(el('span', 'nxc-node__refs', `引用 ${text}`)); + } + return wrap; + } + + /** The node's mode control, found again after the card was rebuilt. */ + function focusModeControl(node) { + const el = dom.nodes.querySelector( + `[data-node-id="${node.id}"] [data-field="mode"] .nxc-dropdown__button`, + ); + el?.focus({ preventScroll: true }); + } + + // ---------------------------------------------------------------- dropdowns + // + // Native + + + + + + +
+ + 100% + + + + + + + + + + + +
+ + +
+
+
+
+ +
+
+ + + +
+ 双击空白处新建图片节点 · Shift+拖拽框选 · 拖节点右侧圆点连线 · 连线即数据流 +
+
+
+ + + + + + + + + + + + + + diff --git a/plugins/antianqi/ai-node-canvas/miniapp/client/mention.mjs b/plugins/antianqi/ai-node-canvas/miniapp/client/mention.mjs new file mode 100644 index 0000000..8d88bf6 --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/miniapp/client/mention.mjs @@ -0,0 +1,96 @@ +// Pure helpers for the inline mention editor. +// +// The caret maths lives here as a plain tree walk so it can be unit tested +// without a browser: `textIndexBefore` only needs `nodeType`, `textContent`, +// `childNodes` and `dataset`, which a tiny fake tree can provide. +// +// A mention chip counts as one character of prompt text. + +const TEXT = 3; +const ELEMENT = 1; + +function isMentionChip(node) { + return node?.nodeType === ELEMENT && Boolean(node?.dataset?.mentionNodeId); +} + +// Number of prompt characters that precede the caret. +// +// `root` is the editor element, `startContainer` the node holding the caret and +// `startOffset` the offset inside it. When the caret sits in a text node, the +// characters of that node before the caret count; the rest of the tree is +// walked in document order. +export function textIndexBefore(root, startContainer, startOffset) { + let index = 0; + const walk = (node) => { + if (node === startContainer) { + if (node.nodeType === TEXT) index += startOffset ?? 0; + return true; + } + if (node.nodeType === TEXT) { + index += (node.textContent ?? '').length; + return false; + } + if (isMentionChip(node)) { + index += 1; + return false; + } + const children = node.childNodes ?? []; + for (const child of children) { + if (walk(child)) return true; + } + return false; + }; + walk(root); + return index; +} + +// Parse an editor's DOM into prompt segments. +// +// A text node contributes a text segment; a mention chip contributes a mention +// segment. Adjacent text segments are merged. +export function parseSegments(editor) { + const segments = []; + for (const child of editor.childNodes ?? []) { + if (child.nodeType === TEXT) { + const text = child.textContent ?? ''; + if (!text) continue; + const last = segments[segments.length - 1]; + if (last && last.type === 'text') last.text += text; + else segments.push({ type: 'text', text }); + } else if (isMentionChip(child)) { + segments.push({ + type: 'mention', + nodeId: child.dataset.mentionNodeId, + kind: child.dataset.mentionKind || 'image', + }); + } + } + return segments; +} + +// Flatten segments back into plain prompt text, as a human would read it. +// A mention renders as `@`. +export function plainText(segments) { + return (segments ?? []) + .map((segment) => (segment.type === 'text' ? segment.text : `@${segment.nodeId}`)) + .join(''); +} + +// Flatten segments into the coordinate space the caret maths uses. +// A mention chip is a single token, so it counts as exactly one character. +// Mixing this representation with `plainText` would misalign every offset. +export const MENTION_PLACEHOLDER = ''; + +export function caretText(segments) { + return (segments ?? []) + .map((segment) => (segment.type === 'text' ? segment.text : MENTION_PLACEHOLDER)) + .join(''); +} + +// Detect an in-progress `@` mention before the caret. +// Returns the query string, or null when there is no mention being typed. +export function detectMention(segments, caretIndex) { + const before = caretText(segments).slice(0, caretIndex); + const match = /@([^\s@]*)$/.exec(before); + return match ? match[1] : null; +} diff --git a/plugins/antianqi/ai-node-canvas/miniapp/client/run-watch.mjs b/plugins/antianqi/ai-node-canvas/miniapp/client/run-watch.mjs new file mode 100644 index 0000000..5767b8c --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/miniapp/client/run-watch.mjs @@ -0,0 +1,98 @@ +// The rules for watching a run. +// +// A run is a thing that happens somewhere else: the runtime asks the +// provider, the page asks the runtime, and the card shows what came back. +// The page has to keep asking, and it has to keep asking even when the +// answer does not come. +// +// Two rules live here, and both of them are about that: +// +// - a request carries a deadline, because a request that never comes back +// would stop the loop that asked for it, and the card would turn for +// ever on a run that is long gone; +// - a run the runtime does not know any more is said out loud, because a +// card that turns for ever tells the user nothing. + +/** + * A call with a deadline. + * + * The call gets an abort signal, so the work it started can be let go when + * the deadline passes: the one that asked is free to ask again, and the + * one that was asked is not left hanging on to a promise nobody waits for. + */ +export async function withDeadline(call, ms) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), ms); + try { + return await call(controller.signal); + } finally { + clearTimeout(timer); + } +} + +/** + * Whether the answer says the run is gone. + * + * The runtime answers 404 for a run it does not know: the runtime was + * restarted, so that run is never going to finish. A request that timed out + * is another matter — the run may still be going — so a timeout is not a + * lost run. + */ +export function runIsGone(error) { + return Number(error?.status) === 404; +} + +/** + * Whether the runtime refused because the node is already running. + * + * That is not a failure: the run that is on the node is the one the chain has + * to wait for, and marking the node as failed would be a lie. + */ +export function isBusyRefusal(error) { + return String(error?.code ?? '') === 'BUSY'; +} + +/** + * Wait for one run to be over. + * + * The run is held on to **by its id**. The page forgets which run belongs to a + * node as soon as that run is over — the node is free to run again — so a + * waiter that looked the run up again would never see it finish, and the chain + * would stand still until its own timeout. + * + * The run that is waited for may be one that was already going: the runtime + * refuses a second run on the same node, so a chain that meets a busy node + * waits for the run that is there. + */ +export function waitForTheRun({ + taskId, + readTask, + timeoutMs = 2_700_000, + intervalMs = 200, + now = () => Date.now(), + schedule = (fn, ms) => setTimeout(fn, ms), +} = {}) { + return new Promise((resolve) => { + const started = now(); + const tick = () => { + const task = taskId ? readTask(taskId) : null; + if (task && !['running', 'waiting-question'].includes(task.status)) { + resolve(task); + return; + } + if (!taskId) { + // There is no run to wait for, so nothing is going to happen. + resolve(null); + return; + } + // A video on MiniMax-H3 takes 15-30 minutes, so the wait is long enough + // for a slow generation and not a timeout of its own. + if (now() - started > timeoutMs) { + resolve(null); + return; + } + schedule(tick, intervalMs); + }; + tick(); + }); +} diff --git a/plugins/antianqi/ai-node-canvas/miniapp/client/styles.css b/plugins/antianqi/ai-node-canvas/miniapp/client/styles.css new file mode 100644 index 0000000..e31e946 --- /dev/null +++ b/plugins/antianqi/ai-node-canvas/miniapp/client/styles.css @@ -0,0 +1,2339 @@ +/* ---------------- semantic tokens ---------------- + The canvas is an independent page and does not inherit Host CSS variables, + so it owns its own palette. `--mcode-*` is the canonical naming; the short + aliases at the bottom are the single adapter layer the component styles + read, so a future Host theme bridge only has to change this block. */ +:root { + color-scheme: light dark; + + --mcode-bg: #ffffff; + --mcode-bg-secondary: #f5f5f5; + --mcode-surface: #ffffff; + --mcode-surface-muted: #fafafa; + --mcode-text: #171717; + --mcode-text-muted: #666666; + --mcode-text-subtle: #adadad; + --mcode-border: #0a0a0a14; + --mcode-border-strong: #0a0a0af2; + --mcode-accent: #0094fc; + --mcode-success: #04b54b; + --mcode-warning: #f56811; + --mcode-danger: #f73646; + --mcode-primary-action-bg: #171717; + --mcode-primary-action-text: #ffffff; + + --mcode-shadow: 0 0 20px rgb(10 10 10 / 8%); + --mcode-shadow-strong: 0 0 20px rgb(10 10 10 / 14%); + --mcode-radius-card: 12px; + --mcode-radius-control: 8px; + --mcode-row-height: 34px; + --mcode-duration-fast: 150ms; + --mcode-duration-panel: 220ms; + --mcode-ease: cubic-bezier(0.22, 0.61, 0.36, 1); + + /* The hues of the data kinds. + A node, its ports and the line between them share one hue, so a chain can + be read by colour alone: amber for text, green for a picture, blue for + video, violet for audio, rose for a style preset. The light values are + dark enough to sit on a white card, the dark ones light enough to sit + on a dark one. */ + --nxc-kind-text: #8f6014; + --nxc-kind-image: #1c7a58; + --nxc-kind-video: #2b62b5; + --nxc-kind-audio: #6b4fbf; + --nxc-kind-style: #b23f78; + + /* adapter: component styles read these short names */ + --bg: var(--mcode-bg); + --bg-panel: var(--mcode-surface); + --line: var(--mcode-border); + --line-soft: var(--mcode-border); + --text: var(--mcode-text); + --text-dim: var(--mcode-text-muted); + --text-faint: var(--mcode-text-subtle); + --brand: var(--mcode-accent); + --brand-2: var(--mcode-accent); + --danger: var(--mcode-danger); + --ok: var(--mcode-success); + --warn: var(--mcode-warning); + --node-radius: var(--mcode-radius-card); +} + +/* The dark palette is written twice on purpose: once for an explicit + data-theme override and once for the system preference, because CSS has no + way to apply a token block conditionally without repeating it. */ +[data-theme='dark'] { + --mcode-bg: #171717; + --mcode-bg-secondary: #1c1c1c; + --mcode-surface: #262626; + --mcode-surface-muted: #262626; + --mcode-text: #ededed; + --mcode-text-muted: #949494; + --mcode-text-subtle: #666666; + --mcode-border: #ffffff14; + --mcode-border-strong: #fffffff2; + --mcode-accent: #0077d9; + --mcode-success: #009c3d; + --mcode-warning: #e25507; + --mcode-danger: #e31937; + --mcode-primary-action-bg: #ffffff; + --mcode-primary-action-text: #171717; + --mcode-shadow: 0 0 20px rgb(0 0 0 / 32%); + --mcode-shadow-strong: 0 0 20px rgb(0 0 0 / 42%); + --nxc-kind-text: #d9a441; + --nxc-kind-image: #35b98a; + --nxc-kind-video: #5a9cf0; + --nxc-kind-audio: #a184f0; + --nxc-kind-style: #e879a8; +} + +@media (prefers-color-scheme: dark) { + :root:not([data-theme='light']) { + --mcode-bg: #171717; + --mcode-bg-secondary: #1c1c1c; + --mcode-surface: #262626; + --mcode-surface-muted: #262626; + --mcode-text: #ededed; + --mcode-text-muted: #949494; + --mcode-text-subtle: #666666; + --mcode-border: #ffffff14; + --mcode-border-strong: #fffffff2; + --mcode-accent: #0077d9; + --mcode-success: #009c3d; + --mcode-warning: #e25507; + --mcode-danger: #e31937; + --mcode-primary-action-bg: #ffffff; + --mcode-primary-action-text: #171717; + --mcode-shadow: 0 0 20px rgb(0 0 0 / 32%); + --mcode-shadow-strong: 0 0 20px rgb(0 0 0 / 42%); + --nxc-kind-text: #d9a441; + --nxc-kind-image: #35b98a; + --nxc-kind-video: #5a9cf0; + --nxc-kind-audio: #a184f0; + --nxc-kind-style: #e879a8; + } +} + +* { + box-sizing: border-box; +} + +/* ---------------- scrollbars ---------------- + Every surface in the canvas stays scrollable with the wheel, but no bar is + ever painted. A Chromium scrollbar on a dark card reads as a stray slab + hanging off the node, and a nested bar inside a node looks worse still. + `scrollbar-width: none` is the rule that actually applies in Chromium; + the ::-webkit rules are the fallback for older WebKit builds. */ +.nxc-app, +.nxc-app * { + scrollbar-width: none; + -ms-overflow-style: none; +} +.nxc-app *::-webkit-scrollbar { + width: 0; + height: 0; + display: none; +} + +html, +body { + height: 100%; + margin: 0; + overflow: hidden; +} + +body { + background: var(--bg); + color: var(--text); + /* The baseline font stack, with the CJK faces after the Latin ones so + Chinese still picks a matching family. */ + font: 14px/1.6 ui-sans-serif, -apple-system, system-ui, "Segoe UI", "PingFang SC", "Microsoft YaHei", Helvetica, Arial, sans-serif; + -webkit-font-smoothing: antialiased; +} + +button, +select, +textarea, +input { + font: inherit; + color: inherit; +} + +.nxc-app { + display: flex; + flex-direction: column; + height: 100%; +} + +/* ---------------- toolbar ---------------- */ + +.nxc-toolbar { + display: flex; + align-items: center; + justify-content: space-between; + gap: 12px; + flex-wrap: wrap; + padding: 9px 12px; + background: var(--mcode-surface); + border-bottom: 1px solid var(--line); + z-index: 30; +} + +.nxc-toolbar__group { + display: flex; + align-items: center; + gap: 7px; + min-width: 0; + /* The toolbar holds more controls than fit in a narrow window. The groups + have to wrap as well, or the last ones run out of the page: the page + itself does not scroll. */ + flex-wrap: wrap; +} + +.nxc-toolbar__group--right { + /* The wrapped lines of the right group stay against the right edge. */ + justify-content: flex-end; +} + +/* The toolbar carries two kinds of verbs: the ones that act on this canvas + and the ones that act on a template. The line between them says where the + canvas ends and a shareable chain begins. */ +.nxc-toolbar__sep { + width: 1px; + height: 16px; + margin: 0 5px; + background: color-mix(in srgb, var(--mcode-text) 22%, transparent); + flex: none; +} + +.nxc-brand { + display: inline-flex; + align-items: center; + gap: 7px; + font-weight: 600; + letter-spacing: 0.3px; + margin-right: 2px; + white-space: nowrap; + /* It is the page heading, but it keeps the look of a label. */ + margin-block: 0; + font-size: inherit; + line-height: inherit; +} + +.nxc-brand__dot { + width: 9px; + height: 9px; + border-radius: 50%; + background: var(--brand); + box-shadow: none; +} + +.nxc-btn { + appearance: none; + border: 1px solid var(--line); + background: var(--mcode-surface-muted); + border-radius: 8px; + /* Baseline: 32-40px control height, 8px corners. */ + min-height: 32px; + padding: 5px 12px; + cursor: pointer; + white-space: nowrap; + transition: background var(--mcode-duration-fast) var(--mcode-ease), border-color var(--mcode-duration-fast) var(--mcode-ease); +} + +.nxc-btn:hover { + background: var(--mcode-bg-secondary); + border-color: var(--mcode-border-strong); +} + +.nxc-btn:active { + background: color-mix(in srgb, var(--mcode-text) 6%, transparent); +} + +.nxc-btn:disabled { + opacity: 0.45; + cursor: not-allowed; +} + +.nxc-btn--primary { + background: var(--mcode-primary-action-bg); + border-color: var(--mcode-primary-action-bg); + color: var(--mcode-primary-action-text); + font-weight: 600; +} + +.nxc-btn--primary:hover { + background: color-mix(in srgb, var(--mcode-primary-action-bg) 85%, transparent); +} + +/* A destructive action carries the danger token the whole time, not only when + the pointer is on it: the button that throws work away should look like it + before it is clicked. */ +.nxc-btn--danger { + border-color: color-mix(in srgb, var(--mcode-danger) 45%, var(--line)); + color: var(--mcode-danger); +} + +.nxc-btn--danger:hover { + border-color: var(--mcode-danger); + background: color-mix(in srgb, var(--mcode-danger) 12%, var(--mcode-surface-muted)); + color: var(--mcode-danger); +} + +/* A small button inside a card or a panel: the same button, less of it. */ +.nxc-btn--small { + min-height: 24px; + padding: 2px 9px; + font-size: 11px; + border-radius: 6px; +} + +.nxc-btn--tiny { + padding: 2px 7px; + font-size: 11px; + border-radius: 6px; +} + +.nxc-select { + appearance: none; + background: var(--mcode-surface-muted); + background-image: linear-gradient(45deg, transparent 50%, currentColor 50%), linear-gradient(135deg, currentColor 50%, transparent 50%); + background-position: right 15px center, right 10px center; + background-size: 5px 5px, 5px 5px; + background-repeat: no-repeat; + padding: 5px 28px 5px 9px; + border: 1px solid var(--line); + border-radius: 8px; + padding: 5px 26px 5px 9px; + cursor: pointer; + max-width: 210px; + text-overflow: ellipsis; +} + +.nxc-select--canvas { + min-width: 140px; +} + +/* The name of the canvas, edited in place. + A native prompt() is a separate OS window that never reaches the screen + inside the MiniApp host, so the name is typed into the page itself. */ +.nxc-rename { + min-width: 140px; + max-width: 210px; + border: 1px solid var(--brand); + border-radius: 8px; + padding: 5px 9px; + background: var(--mcode-surface); + color: var(--text); + font: inherit; + font-size: 12px; +} + +.nxc-rename:focus-visible { + outline: 2px solid var(--brand); + outline-offset: 1px; +} + +/* ---------------- asking in the page ---------------- + A native confirm() is a separate OS window that never reaches the screen + inside the MiniApp host, so the question is drawn here instead. */ +.nxc-confirm { + position: fixed; + inset: 0; + z-index: 60; + display: flex; + align-items: center; + justify-content: center; + background: rgb(0 0 0 / 42%); +} + +/* The rule above says how the question is drawn, not that it is always + there: a `display` on the class would beat the `hidden` attribute. */ +.nxc-confirm[hidden] { + display: none; +} + +.nxc-confirm__box { + max-width: 360px; + padding: 20px 22px 18px; + border: 1px solid var(--line); + border-radius: 12px; + background: var(--mcode-surface); + color: var(--text); + box-shadow: var(--mcode-shadow-strong); +} + +.nxc-confirm__text { + margin: 0 0 14px; + font-size: 13px; + line-height: 1.6; +} + +.nxc-confirm__actions { + display: flex; + gap: 8px; + justify-content: flex-end; +} + +/* ---------------- self-drawn dropdown ---------------- + A native