Summary
I'm picking up token optimization next, and I'd rather agree the approach here than open a PR and find out we wanted different things. The obvious version of this task — make the prompts shorter — is also the version most likely to quietly make generated activities worse.
The concern that started this was that token optimization shouldn't cost us activity quality. Agreed, and the awkward part is that today we can't tell either way. There's no eval harness, no CI, and sampling temperature sits at 0.2–0.3 across the providers, so the same prompt doesn't produce the same activity twice. "I tried it and it still worked" isn't evidence and I don't want to hand you a PR resting on it.
Baseline
Measured on 18d566e, FractionMatch-style spec, no retrieved references: plan system 2,683 tok, plan user 164, codegen system 6,001, codegen user 268 — about 9,100 input tokens per generation, with retrieved references adding on top. Two thirds of that is the codegen system prompt.
Most of this is cacheable, and caching can't change output
There's no prompt caching in llm/providers.py for any provider. That's the biggest thing on the table, and it's the one piece of this work that carries no quality risk at all.
get_api_reference() and get_ui_design_reference() return static module constants — 6,092 chars identical on every job for every user. More usefully, both prompts already open with long runs of fixed text before anything spec-dependent appears. I measured where each one first diverges by building them across specs that differ in category, template, licence, prompt and plan:
- Plan system prompt: the first 8,507 chars (~2,126 tok) are byte-identical across all nine variants I tried. Everything spec-specific starts at
OPTIONAL DISCOVERY DIRECTION.
- Codegen system prompt: the first 16,132 chars (~4,033 tok) are identical — 67% of that prompt.
So about 68% of input tokens per generation sit in a stable prefix today, with no text moved anywhere. A cache breakpoint at the end of each run is not a rewrite; the model receives byte-identical input, so output cannot change. Not "probably won't" — cannot. Both prefixes clear the ~1,024-token minimum that caching needs, and cached reads bill at roughly a tenth. Each provider needs its own native form of this (Anthropic takes an explicit marker on the system block, OpenAI caches automatically above the same threshold, Gemini has implicit caching), so it's a per-provider change rather than one shared switch.
One thing worth being precise about: this cuts cost, not token count. If the goal is a smaller context rather than a smaller bill, caching does nothing for us.
One reordering would buy a lot more
While locating those boundaries I found something that looks unintentional. In the codegen system prompt the first thing that varies is the code_size length instruction, at char 16,132. Both static reference blocks land after it — api_reference at 17,807 and ui_design_reference at 18,823.
That ordering is what stops 6,092 chars of permanently-fixed reference material from being cacheable, because a cache prefix ends at the first byte that changes. Moving that one length line below the references would pull roughly another 1,523 tok into the stable prefix and take us from ~68% to ~84% of input cacheable.
That one does move text, so unlike the breakpoints it isn't free — but it relocates a single sentence about output length rather than touching any reference material, which is about the smallest semantic change available.
The part that genuinely risks quality
The same 6,092 chars go into both the plan call and the codegen call. That's paid twice per job, but they're two separate requests rather than duplication inside one, so dropping it from the plan prompt really does remove information the planner sees. It's a candidate, not a free win.
The piece I'd want to test: api_reference is code-level guidance — subclass sugar3.activity.activity.Activity, wire the toolbar, implement write_file/read_file. The planner emits JSON metadata (ui_regions, features, state_schema) and never writes a line of Python. Plausibly it's doing nothing in the plan call. It's 974 chars, so it's a cheap and clean hypothesis — but I'd rather measure it than argue it. Trimming ui_design_reference (5,118 chars, in every codegen prompt) is a much bigger swing and I wouldn't touch it without numbers.
What's missing
We already have three automatic accept/reject gates in validator.py, runtime_check.py and critic.py. That's a better starting point than most projects get, because they produce a pass/fail per generation with no human judgement needed.
What doesn't exist is anything that runs them across a fixed set of specs and reports aggregate pass rates. That's what I'd build first: a handful of specs spanning the categories, run repeatedly per config, recording per-gate pass rate, repair attempts consumed, input and output tokens, and wall time. Then a prompt change ships only if pass rate holds and tokens drop. Because temperature isn't zero it has to be repeated runs per spec, not a single before-and-after.
Building that before touching prompts is the slower path, but it's the only one where "did this hurt quality?" has a number behind it — and it stays useful for every prompt change after this one, not just mine.
Separately
While measuring I found code_size doesn't do what its name suggests. Compact versus full changes the input prompt by 0.6% (5,993 vs 6,028 tok) because _CODE_SIZE_TOKENS only sets max_output_tokens. Someone choosing "compact" to keep costs down saves essentially nothing on input. That's a UX bug rather than a token optimization, so I'll file it on its own unless you'd rather it rode along here.
What I need from you
- Cost or context? If the goal is a smaller bill, caching plus the reorder is most of the win and it's close to risk-free. If it's genuinely fewer tokens — context pressure, smaller or local models — then the trimming work is unavoidable and the harness isn't optional.
- Harness first, or caching first? Caching is the one piece that's safe to ship without it, so I'm happy to do that first and build the harness before anything that removes text.
- Running the harness means real API calls against a real key. Is there a budget or a key I should use, or should I run it against a local provider and accept the results won't fully transfer to the cloud ones?
Happy to start on any of it — I just don't want to guess at the first one and build the wrong thing.
Summary
I'm picking up token optimization next, and I'd rather agree the approach here than open a PR and find out we wanted different things. The obvious version of this task — make the prompts shorter — is also the version most likely to quietly make generated activities worse.
The concern that started this was that token optimization shouldn't cost us activity quality. Agreed, and the awkward part is that today we can't tell either way. There's no eval harness, no CI, and sampling temperature sits at 0.2–0.3 across the providers, so the same prompt doesn't produce the same activity twice. "I tried it and it still worked" isn't evidence and I don't want to hand you a PR resting on it.
Baseline
Measured on
18d566e,FractionMatch-style spec, no retrieved references: plan system 2,683 tok, plan user 164, codegen system 6,001, codegen user 268 — about 9,100 input tokens per generation, with retrieved references adding on top. Two thirds of that is the codegen system prompt.Most of this is cacheable, and caching can't change output
There's no prompt caching in
llm/providers.pyfor any provider. That's the biggest thing on the table, and it's the one piece of this work that carries no quality risk at all.get_api_reference()andget_ui_design_reference()return static module constants — 6,092 chars identical on every job for every user. More usefully, both prompts already open with long runs of fixed text before anything spec-dependent appears. I measured where each one first diverges by building them across specs that differ in category, template, licence, prompt and plan:OPTIONAL DISCOVERY DIRECTION.So about 68% of input tokens per generation sit in a stable prefix today, with no text moved anywhere. A cache breakpoint at the end of each run is not a rewrite; the model receives byte-identical input, so output cannot change. Not "probably won't" — cannot. Both prefixes clear the ~1,024-token minimum that caching needs, and cached reads bill at roughly a tenth. Each provider needs its own native form of this (Anthropic takes an explicit marker on the system block, OpenAI caches automatically above the same threshold, Gemini has implicit caching), so it's a per-provider change rather than one shared switch.
One thing worth being precise about: this cuts cost, not token count. If the goal is a smaller context rather than a smaller bill, caching does nothing for us.
One reordering would buy a lot more
While locating those boundaries I found something that looks unintentional. In the codegen system prompt the first thing that varies is the
code_sizelength instruction, at char 16,132. Both static reference blocks land after it —api_referenceat 17,807 andui_design_referenceat 18,823.That ordering is what stops 6,092 chars of permanently-fixed reference material from being cacheable, because a cache prefix ends at the first byte that changes. Moving that one length line below the references would pull roughly another 1,523 tok into the stable prefix and take us from ~68% to ~84% of input cacheable.
That one does move text, so unlike the breakpoints it isn't free — but it relocates a single sentence about output length rather than touching any reference material, which is about the smallest semantic change available.
The part that genuinely risks quality
The same 6,092 chars go into both the plan call and the codegen call. That's paid twice per job, but they're two separate requests rather than duplication inside one, so dropping it from the plan prompt really does remove information the planner sees. It's a candidate, not a free win.
The piece I'd want to test:
api_referenceis code-level guidance — subclasssugar3.activity.activity.Activity, wire the toolbar, implementwrite_file/read_file. The planner emits JSON metadata (ui_regions,features,state_schema) and never writes a line of Python. Plausibly it's doing nothing in the plan call. It's 974 chars, so it's a cheap and clean hypothesis — but I'd rather measure it than argue it. Trimmingui_design_reference(5,118 chars, in every codegen prompt) is a much bigger swing and I wouldn't touch it without numbers.What's missing
We already have three automatic accept/reject gates in
validator.py,runtime_check.pyandcritic.py. That's a better starting point than most projects get, because they produce a pass/fail per generation with no human judgement needed.What doesn't exist is anything that runs them across a fixed set of specs and reports aggregate pass rates. That's what I'd build first: a handful of specs spanning the categories, run repeatedly per config, recording per-gate pass rate, repair attempts consumed, input and output tokens, and wall time. Then a prompt change ships only if pass rate holds and tokens drop. Because temperature isn't zero it has to be repeated runs per spec, not a single before-and-after.
Building that before touching prompts is the slower path, but it's the only one where "did this hurt quality?" has a number behind it — and it stays useful for every prompt change after this one, not just mine.
Separately
While measuring I found
code_sizedoesn't do what its name suggests. Compact versus full changes the input prompt by 0.6% (5,993 vs 6,028 tok) because_CODE_SIZE_TOKENSonly setsmax_output_tokens. Someone choosing "compact" to keep costs down saves essentially nothing on input. That's a UX bug rather than a token optimization, so I'll file it on its own unless you'd rather it rode along here.What I need from you
Happy to start on any of it — I just don't want to guess at the first one and build the wrong thing.