English · 简体中文

chumi-promo.mp4	chumi-pet.mp4
With music · 中文 · faster cuts to download: 59 s, 45 s	Chumi.app's desktop pet · how it works · 中文

Chumi listens to academic meetings with you, explains what you missed, and hands what was decided to your coding agent.

It transcribes lab meetings, advisor 1:1s, lectures and seminars on your Mac, in English, Mandarin, or both at once.

• During the meeting: catch up in one click, get terms explained at your level, and ask about the slide on screen. Every answer cites the moment it came from. A desktop pet can sit beside your call and do the same from its toolbar.
• After the meeting: read a structured summary, play the recording back from any line, and review a handoff of decisions and actions.
• In your project: Codex app, Claude Code, Codex, opencode, Qwen Code, Antigravity or DeepSeek Harness continues the work, reading the original evidence through Chumi's MCP server.
• At the next meeting: answers can draw on what those agents already did.

It runs on the ChatGPT or Claude plan you already have, or on a local model. By default, audio never leaves your Mac.

Contents: See it in action · Install · Choose an AI · Privacy · The agent loop · All features · Development

See it in action

Real screens from a synthetic NLP lab meeting: Maya presents a 7B model trained with DPO to cite its sources, and the lab argues about whether its gain is real or a quirk of the LLM judge. Scroll sideways for all nine →

Lost the thread? Catch me up. What you missed and what is being argued now, over the last 2 or 5 minutes or the whole meeting. Every point links to its lines.

Explained at your level Select a term like KV cache: the intuition, how it was used here and why it matters, pitched to your background. In English or 中文.

Ask about the slide on screen Capture the Zoom share or a lecture video. Chumi keeps one picture per slide and weighs it against what was actually said.

A summary you can trust Shaped for the meeting type. Preliminary results stay preliminary, disagreements are kept, and owners appear only if someone named them.

Listen again The ▶ beside a line plays from there, and the line being heard stays highlighted. Export audio saves the meeting as one file.

A handoff you review Decisions and actions, each cited and labelled by origin. Confirm, edit or reject each one, then Continue in Codex app or Claude Code.

The next meeting remembers Turn on Project context and answers also cite your agents' earlier sessions, found through mybot, as [P1], [P2].

Get Chumi ready Home walks you through AI sign-in, your background and the microphone. Then ⚡ Quick start, or ⌃⌥⌘L from any app.

Speech stays on your Mac Local whisper.cpp, with a model picked for your Mac. Cloud speech recognition only if you choose it.

_{About the screenshots: real local transcription, real AI answers, and real agent sessions that continued the handoff. The meeting is a written script with fictional people, voiced by macOS text-to-speech (samples/llm-meeting-en; the Chinese set uses the bilingual samples/llm-meeting), and the profile is fictional too. Nothing comes from real meetings.}

Install

You need a Mac with Apple Silicon on macOS 14.4 or later, and a ChatGPT or Claude subscription. Without one, a Mac with 16 GB of memory or more can run a free local model.

Open Terminal and paste:

curl -fsSL https://raw.githubusercontent.com/FakeEnd/chumi/main/scripts/install.sh | bash

This downloads Chumi into ~/Chumi. The guided setup takes 5 to 15 minutes. It installs what is missing (asking first), sets up local speech recognition with a model that suits your Mac, builds Chumi.app, and helps you sign in to an AI. Your sign-in stays with the official tool; Chumi never sees it.

On first launch, the Get Chumi ready checklist on Home covers the rest: AI sign-in, your background and microphone access. Then click ⚡ Quick start, or press ⌃⌥⌘L from any app. Press it again to pause.

• By hand: git clone https://github.com/FakeEnd/chumi.git && cd chumi && bash scripts/setup.sh. The install guide covers the options (including --mirror for mainland China), permissions, troubleshooting and uninstalling.
• Check an install: npm run doctor prints the fix for anything missing.
• In a browser: npm start, then open http://127.0.0.1:4777.
• Windows (desktop preview): an installer with microphone and computer-audio capture, local speech recognition, Codex and the desktop pet. It includes the shared recording, handoff and project-progress workflows. See docs/windows.md for Windows 11 verification and build instructions.
• iPhone (prototype): a SwiftUI app in ios/ records and transcribes on the phone with Apple Speech, and pairs with Chumi on the Mac for AI help. It needs full Xcode and hasn't yet been compiled or tested on a device. See the iPhone guide.

Choose an AI

Pick one in Settings → AI provider. Chumi uses exactly one at a time and never switches to a paid API on its own.

AI	Sign-in	What you pay
ChatGPT plan, through OpenAI's Codex app-server	codex login	Included plan usage. Purchased credits only if you allow them.
Claude plan, through the unmodified claude -p	claude auth login	Plan usage limits. Chumi stops at paid overage.
Local model (Ollama, LM Studio)	none	Free, on your Mac. Chumi suggests models that fit its memory.
Compatible API (Qwen, Doubao, MiniMax, GLM, custom)	key in macOS Keychain	Your provider's billing; off until you enable it.
Anthropic API key	key in macOS Keychain	Pay per request; off until you enable it.

Local models and compatible APIs are set up in docs/other-models.md. The reasoning behind these choices, with links to the official terms, is in docs/provider-research.md.

Speech recognition is separate from the AI. It runs locally with whisper.cpp by default, and Settings → Transcription recommends one of seven models for your Mac. ElevenLabs Scribe and Qwen3-ASR (Alibaba Cloud or self-hosted) are opt-in alternatives that send meeting audio off the Mac; if one fails, the local model takes over. See docs/transcription.md.

Privacy

• Audio stays on your Mac with the default local speech recognition. Recordings are stored only on this computer, and deleting a session deletes them.
• Everything is local: sessions, transcripts, answers and your profile live in a SQLite database on your Mac. The server listens only on 127.0.0.1.
• What goes to the AI is the text it needs (and slide images, if you turn that on). Settings → Privacy & data explains exactly what is sent and when.
• Agents only read: the MCP server is read-only, only confirmed handoff items are requested work, and the agent proposes a plan before changing anything.

The agent loop

flowchart LR
  M["Meeting<br/>live transcript · catch-up<br/>explanations at your level"] --> H["Reviewed handoff<br/>decisions · actions · open questions<br/>each cited to the transcript"]
  H --> A["Your coding agent<br/>Codex app · Claude Code · Codex · opencode · Qwen Code · Antigravity · DeepSeek Harness<br/>reads the evidence over MCP"]
  A --> P["Project memory<br/>the agents' sessions,<br/>indexed by mybot"]
  P -- "Project context" --> M

Loading

To set it up:

1. Assign the meeting to a project, the folder you work in. Pick it when you start, or later in the top bar. Nothing is assigned automatically.
2. Choose where to continue. Codex app is the default and needs no setup: Continue opens a new chat with the handoff in its composer, and nothing runs until you click Send. To let terminal agents fetch the original passages too, connect them once in Settings → Coding agents; Test a real agent read confirms the agent can reach Chumi's read-only MCP server.
3. Optional: run mybot locally for Project context. Its address is in Settings → Integrations (default http://127.0.0.1:8788). Nothing is searched until you turn Project context on for a meeting.

If the meeting changes after you confirm (a regenerated summary, a resumed recording), Chumi and the agent both see that the handoff may need another look. Details: docs/agents.md and docs/mybot.md.

All features

Show the full list

Listening

• Microphone for in-person meetings, computer audio for Zoom, Teams or recorded lectures (headphones are fine), or both for hybrid meetings, with echo cancellation and duplicate filtering.
• English, Mandarin and code-switching. Vocabulary hints improve the spelling of jargon and names.
• Pause, resume, a global quick-start shortcut, level meters and clear permission guidance.
• Compact view (⌘⇧., or Ctrl+Shift+. on Windows) hides the header and toolbar, leaving the summary, transcript and assistant.
• Optional speaker labels after the meeting (npm run setup:speakers): voices are grouped locally as A/B/C, and you name and correct them. See docs/speaker-diarization.md.
• A desktop pet in Chumi.app: its toolbar starts, pauses and ends a meeting, catches you up and takes a question, and continues a confirmed handoff. Clicks elsewhere pass through, and it stays out of screen shares. See docs/desktop-pet.md.

Understanding

• Catch me up over the last 2 minutes, the last 5 minutes or the whole session.
• Questions about a selected passage, the surrounding discussion and relevant earlier passages.
• Explanation depth (Quick, Standard, Deep) and, on supported providers, reasoning effort (Low to Max).
• Optional auto-explain cards for unfamiliar terms, and a running overview with a topic timeline.
• Answers form a timeline marked with the meeting time each was asked at. Only the latest stays open; the rest fold to one line and open with a click. Each question is tagged as about the meeting or a tangent.
• Answers keep what the speakers said, background added by Chumi and interpretation and uncertainty apart. If a speaker points at a slide Chumi can't see, it says so.

Remembering

• Session types (rename or add your own in Settings → Session types), each with its own summary style.
• History with full-text search in English and 中文, reopen, Markdown or plain-text export, and delete.
• Audio recording per session: every source mixed into one track in transcript time, with long pauses shortened to a one-second break. Export gives one .m4a on macOS (about 14 MB per hour) or a .wav on Windows. The original takes use about 115 MB per hour per source.
• Transcript lines are saved as they arrive. After a crash, the session reopens with a gap marker, its recorded audio is kept, and it can be resumed.
• mybot in both directions: Project context reads your agents' sessions, and finished meetings can be exported into mybot's memory (owner-only by default). See docs/mybot.md.

Cost control

• Billing is shown, with quota when the provider reports it. Subscription adapters pause when included usage runs out.
• Background work (overview, auto-explain) uses low effort where supported, and long sessions use rolling notes instead of resending the whole transcript.

How it works

Chumi.app (Swift window + macOS privacy identity)
├── chumi-audio   AVAudioEngine mic (+ echo cancellation) · Core Audio process tap
├── chumi-screen  ScreenCaptureKit window capture
├── chumi-hotkey  global quick-start shortcut
└── Node.js server (TypeScript, node:sqlite)
    ├── voice-activity segmenter → transcriber → whisper-server (local) or an opt-in speech service
    ├── SQLite: sessions · raw transcript · gaps · AI outputs (kept separate) · screen captures
    ├── recordings: timed takes per source, mixed on the fly for playback and export
    ├── assistant: catch-up · Q&A · explain · overview · auto-explain · summary · handoff
    ├── MCP server (read-only) for coding agents
    └── providers: Codex app-server │ claude -p │ Anthropic API │ compatible local server / API
React UI ⇄ HTTP + Server-Sent Events on 127.0.0.1

Development

npm run check                                  # type checks and unit tests
node scripts/acceptance.ts                     # end to end: a 20-minute sample through real Whisper, offline AI stand-in
node scripts/check-audio.ts system 6 --play    # real computer-audio capture (use "mic 6" for the microphone)
npm run build:tools                            # rebuild the window-capture and shortcut helpers

Regenerating the screenshots and video

1. Render the sample meetings with macOS say (about a minute each): node scripts/make-sample.mjs --script samples/llm-meeting-en/script.json, and the same for samples/llm-meeting.
2. Agent loop, once: node scripts/demo-shoot.ts --prepare-agent --project-folder DIR drafts and confirms a handoff for a project folder and keeps its data folder. Run Claude Code (or Codex) in DIR with that prompt and Chumi's MCP server pointed at the kept data folder, and let mybot index the session.
3. Screenshots: node scripts/demo-shoot.ts --project-folder DIR (add --lang zh for the Chinese set in docs/assets/screens/zh/). It uses about 15 requests on your Claude plan, or on your ChatGPT plan with --provider codex.
4. Live footage: node scripts/record-footage.ts (and --lang zh) records the real window and desktop pet while the answers stream in, into dist/footage/. It rebuilds the web UI first, then uses about 6 requests (Codex by default; --provider claude-cli for your Claude plan). --only agent --project-folder DIR records continuing a confirmed handoff (nothing is actually opened) and keeps its data folder for a real agent run in DIR.
5. Video: node scripts/make-video.ts renders docs/video/promo.html; add --lang zh, or --speed 1.5 / --speed 2 for the faster cuts. node scripts/make-pet-video.ts renders the desktop-pet video (--lang en for English). Both need the footage from step 4. Then bash scripts/make-banner.sh re-renders the banner and the two video cards, and makes each card its video's first frame. Upload the four tour and pet MP4s to GitHub (drag them into a comment) and put their links on the cards: GitHub shows them as players.

Path	Contents
server/	Audio pipeline, speech recognition, recordings, assistant, AI providers, MCP server, storage
web/	React interface
native/	Swift app shell and helpers (native/README.md), and the Windows desktop app
scripts/	Setup, acceptance test, audio check, screenshots, video, sample generator
samples/	Synthetic meetings: llm-meeting (screenshots, video) and lab-meeting (acceptance tests)
integrations/mybot/	mybot source adapter, patch, installer and test
docs/	Guides, provider research and the verification log

Status

Chumi is a working personal tool. What has been verified, and how, is in docs/verification.md.

Known limits: speaker labels run after the meeting and don't identify real people; protected (DRM) video captures as black; Claude reports plan usage only after each request; the app is ad-hoc signed, so rebuilds ask for permissions again.

Next: paper and slide-deck files as context, live speaker labels, a cross-session glossary, spoken explanations, a menu-bar mini mode, and a per-session usage meter.

Related: mybot, one searchable memory for all your Claude Code and Codex conversations; nebula-notebook, an agent-native notebook.

License

MIT. See LICENSE. Speech models are downloaded from the whisper.cpp project (MIT) and are not part of this repository.