FE-1834: Teach Brunch voice agent custom words by kostandinang · Pull Request #9899 · hashintel/hash

kostandinang · 2026-10-02T13:42:39Z

🌟 What is the purpose of this PR?

Let people teach Brunch names and terms through a default-off Custom words Labs setting. Saved spellings and optional pronunciation hints help the assistant recognize and say those words in Voice conversations.

🔗 Related links

FE-1834 (internal)
FE-1828: Add interview length controls to Brunch #9894 uses the same version-two contextual envelope; see Known issues.

🚫 Blocked by

None.

🔍 What does this change?

Add a Voice-only Words header button and a Custom words dialog for adding, editing and removing up to 50 entries.
Store words in this browser, per principal and conversation, and clear them with the conversation. Unsaved edits survive another tab deleting the word.
Send spellings to Brunch and both voice transcription prompts, and pronunciation hints to the speaking model. Transcription models and semantic VAD are unchanged.
Keep a running Voice session on the hints it started with, including retries, until it restarts.
The word list and limits live in the website's Voice code; the spelling bounds and the words envelope live in the SDCPN plugin. Brunch core, Petrinaut Core and the stock assistant prompt are unchanged.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

Modifies @hashintel/petrinaut (assistant header controls) and includes a changeset.

📜 Does this require a change to the docs?

User guide at apps/petrinaut-website/docs/custom-words.md.

🕸️ Does this require a change to the Turbo Graph?

Does not affect the execution graph.

⚠️ Known issues

One Live Voice turn was verified with real microphone and speaker audio; legacy Realtime and physical speaker echo were not tested.
The 50-entry limit is also bounded by spelling, pronunciation, total-text and header size limits.
FE-1828: Add interview length controls to Brunch #9894 also defines petrinaut-contextual-user-message:v2, with a submissionContext payload instead of words. Whichever merges second must move onto the other's envelope.

🛡 What tests cover this?

SDCPN plugin, Brunch core, Petrinaut Core, the Petrinaut assistant panel, apps/brunch-agent and the website's Voice and Brunch suites pass; type, lint and format checks pass.
Coverage includes validation, browser persistence, cross-tab edits, envelope encoding, the model seeing only the user's own words when history is reopened, prompt wiring, session snapshots and echo filtering.

❓ How to test this?

Enable Use Brunch and Custom words in User settings → Labs.
In Voice, open Words and add a name with an optional pronunciation; edit, remove and reload to check persistence. The button is hidden in text mode.
Start Voice and say the name. Change its hint mid-session, then restart Voice to apply it.
Clear the conversation and confirm its words are gone.

📹 Demo

custom-words-live-demo.mp4

vercel · 2026-10-02T13:42:46Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Oct 14, 2026 6:22am UTC
hashdotdesign-tokens	Ready	Preview	Oct 14, 2026 6:22am UTC
petrinaut	Ready	Preview	Oct 14, 2026 6:22am UTC
petrinaut-docs	Ready	Preview	Oct 14, 2026 6:22am UTC

cursor · 2026-10-02T13:59:41Z

PR Summary

Medium Risk
Changes user-message framing, model instructions, and Voice provider prompts; bounded validation and display stripping reduce prompt-injection risk, but custom terms still flow to Brunch and OpenAI.

Overview
Adds Custom words for Brunch Voice: a default-off Labs toggle, a Voice-only Words header control (via new @hashintel/petrinaut headerActions / actionsInputMode), and a per-conversation browser-stored list of spellings plus optional pronunciation notes.

Host → Brunch: Active spellings are captured per user turn and sent in a petrinaut-contextual-user-message:v2 envelope; the chat agent persists the latest snapshot and injects spellings as untrusted model instructions while context projection and UI/transcript paths show only human userText. Voice: One words snapshot is pinned for each session (including retries) and passed to server handlers that extend transcription prompts (spellings) and speaking instructions (pronunciation only), with shadow filtering for transcription-prompt echo.

Validation, encoding, and spelling limits live in the SDCPN plugin and shared voice-words helpers; clearing the conversation drops the local list.

^{Reviewed by Cursor Bugbot for commit b24e2cf. Bugbot is set up for automated code reviews on this repo. Configure here.}

cursor

Stale Bugbot comment from a previous run.

The approval coordinator now reports refused calls, so Live progress treats a denied or stopped edit as finished while its refusal output is still on its way back, instead of reading the stale input-available part as executing. Co-authored-by: Cursor <cursoragent@cursor.com>

…kostandin/fe-1834-teach-brunch-your-words Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx # apps/petrinaut-website/src/main/app/voice-interview/live-conversation-control.tsx

…ch-brunch-your-words Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx # apps/petrinaut-website/src/main/app/voice-interview/live-conversation-control.test.tsx # apps/petrinaut-website/src/main/app/voice-interview/live-conversation-control.tsx

…ch-brunch-your-words

…ch-brunch-your-words # Conflicts: # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx # apps/petrinaut-website/src/main/app/plugins/voice/live/live-brunch-bridge/utterance-pipeline.ts # apps/petrinaut-website/src/main/app/plugins/voice/live/live-conversation.ts # apps/petrinaut-website/src/main/app/plugins/voice/realtime/openai-realtime-session.ts # apps/petrinaut-website/src/main/app/plugins/voice/session/voice-interview-control.tsx # apps/petrinaut-website/src/main/app/plugins/voice/shared/classify-interruption.test.ts

…ch-brunch-your-words Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx

Co-authored-by: Cursor <cursoragent@cursor.com>

…ch-brunch-your-words Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # apps/brunch-agent/src/ui/chat.tsx # apps/petrinaut-website/README.md # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx # apps/petrinaut-website/src/main/app/plugins/brunch/brunch-panel-transport.ts # libs/@hashintel/brunch-agent/packages/plugin-sdcpn/src/contextual-user-message.ts # libs/@hashintel/brunch-agent/packages/transport-aisdk/src/index.ts # libs/@hashintel/brunch-agent/packages/transport-aisdk/src/transcript.ts # libs/@hashintel/brunch-agent/packages/transport-aisdk/test/chat-transport.test.ts # libs/@hashintel/brunch-agent/packages/transport-aisdk/vite.config.ts # libs/@local/flue-aisdk-transport/package.json # libs/@local/flue-aisdk-transport/src/client/contextual-user-message.test.ts # libs/@local/flue-aisdk-transport/src/client/words.test.ts

…ch-brunch-your-words

lunelson · 2026-10-15T14:16:57Z

Closing in favor of #10054

github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/tests New or updated tests area/apps labels Oct 2, 2026

vercel Bot deployed to Preview – petrinaut October 2, 2026 13:46

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 2, 2026 13:46

View deployment

github-advanced-security Bot found potential problems Oct 2, 2026

View reviewed changes

Comment thread libs/@hashintel/brunch-agent/packages/core/src/words.ts Fixed

kostandinang changed the title ~~FE-1834: Add Custom words for Voice~~ FE-1834: Teach Brunch your words Oct 2, 2026

vercel Bot deployed to Preview – hash October 2, 2026 13:52

View deployment

kostandinang marked this pull request as ready for review October 2, 2026 13:59

vercel Bot deployed to Preview – petrinaut October 2, 2026 13:59

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 2, 2026 14:01

View deployment

github-advanced-security Bot found potential problems Oct 2, 2026

View reviewed changes

Comment thread libs/@hashintel/brunch-agent/packages/transport-aisdk/src/words.ts Fixed

kostandinang changed the title ~~FE-1834: Teach Brunch your words~~ FE-1834: Teach Brunch voice agent custom words Oct 2, 2026

vercel Bot deployed to Preview – hash October 2, 2026 14:06

View deployment

vercel Bot deployed to Preview – petrinaut October 2, 2026 14:09

View deployment

github-advanced-security Bot found potential problems Oct 2, 2026

View reviewed changes

Comment thread libs/@hashintel/brunch-agent/packages/transport-aisdk/src/words.ts Fixed

vercel Bot deployed to Preview – petrinaut-docs October 2, 2026 14:11

View deployment

vercel Bot deployed to Preview – petrinaut October 2, 2026 14:14

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 2, 2026 14:16

View deployment

kostandinang mentioned this pull request Oct 2, 2026

FE-1828: Add interview length controls to Brunch #9894

Closed

3 of 4 tasks

kostandinang force-pushed the kostandin/fe-1834-teach-brunch-your-words branch from 865fe2c to 9c8d16b Compare October 2, 2026 14:23

vercel Bot deployed to Preview – hash October 2, 2026 14:27

View deployment

vercel Bot deployed to Preview – petrinaut October 2, 2026 14:27

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 2, 2026 14:28

View deployment

cursor Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread apps/petrinaut-website/src/main/app/local-storage-demo/conversation-words.ts Resolved

vercel Bot deployed to Preview – hash October 6, 2026 14:19

View deployment

lunelson previously approved these changes Oct 7, 2026

View reviewed changes

kostandinang dismissed lunelson’s stale review via d257869 October 7, 2026 09:33

vercel Bot deployed to Preview – petrinaut October 7, 2026 09:36

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 7, 2026 09:38

View deployment

Base automatically changed from kd/fe-1833-speak-progress to main October 7, 2026 11:28

vercel Bot had a problem deploying to Preview – petrinaut-docs October 7, 2026 14:12 Failure

vercel Bot deployed to Preview – hashdotdesign-tokens October 7, 2026 14:13

View deployment

vercel Bot deployed to Preview – petrinaut October 7, 2026 14:14

View deployment

vercel Bot deployed to Preview – hash October 7, 2026 14:22

View deployment

lunelson previously approved these changes Oct 8, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into kostandin/fe-1834-tea…

19edf24

…ch-brunch-your-words

vercel Bot deployed to Preview – hashdotdesign-tokens October 8, 2026 08:27

View deployment

vercel Bot deployed to Preview – petrinaut October 8, 2026 08:27

View deployment

vercel Bot deployed to Preview – petrinaut-docs October 8, 2026 08:30

View deployment

vercel Bot deployed to Preview – hash October 8, 2026 08:34

View deployment

kostandinang and others added 3 commits October 8, 2026 16:11

Merge remote-tracking branch 'origin/main' into kostandin/fe-1834-tea…

03b27e6

…ch-brunch-your-words Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # apps/petrinaut-website/src/main/app/local-storage-demo/local-storage-demo-app.tsx

Format voice-interview-control import

d763a50

Co-authored-by: Cursor <cursoragent@cursor.com>

lunelson previously approved these changes Oct 9, 2026

View reviewed changes

kostandinang mentioned this pull request Oct 9, 2026

FE-1766: Make the chat transport break loudly when Flue or the AI SDK drifts #9854

Merged

3 tasks done

github-advanced-security Bot found potential problems Oct 13, 2026

View reviewed changes

Comment thread libs/@hashintel/brunch-agent/packages/plugin-sdcpn/src/words.ts Dismissed

Merge remote-tracking branch 'origin/main' into kostandin/fe-1834-tea…

b24e2cf

…ch-brunch-your-words

This was referenced Oct 15, 2026

FE-1828: Add interview length controls to Brunch #10053

Merged

FE-1834: Teach Brunch voice agent custom words #10054

Merged

Navigation Menu

Repository navigation

FE-1834: Teach Brunch voice agent custom words - #9899

FE-1834: Teach Brunch voice agent custom words#9899
kostandinang wants to merge 30 commits into
mainfrom
kostandin/fe-1834-teach-brunch-your-words

kostandinang commented Oct 2, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

cursor Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

lunelson commented Oct 15, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Conversation

kostandinang commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌟 What is the purpose of this PR?

🔗 Related links

🚫 Blocked by

🔍 What does this change?

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

📜 Does this require a change to the docs?

🕸️ Does this require a change to the Turbo Graph?

⚠️ Known issues

🛡 What tests cover this?

❓ How to test this?

📹 Demo

Uh oh!

vercel Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

cursor Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lunelson commented Oct 15, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

kostandinang commented Oct 2, 2026 •

edited

Loading

vercel Bot commented Oct 2, 2026 •

edited

Loading

cursor Bot commented Oct 2, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading