Repository navigation
Cache the system prompt and price cached tokens - #13
Merged
Merged
Conversation
Every model call now sends the system prompt as one text block with a cache_control ephemeral breakpoint (default 5 minute TTL). The system prompt depends only on the envelope config and the building names, so it is byte-identical every hour of a run; the hour's observation follows it in the user message. The manifest records prompt_cache, and replay reads it (absent means off), so runs recorded before caching rebuild the exact request they sent and still replay. Cost accounting reads cache_read_input_tokens and cache_creation_input_tokens from every response. input_tokens is now uncached input only. The model table carries cache write (1.25x input) and cache read (0.1x input) prices, and the run budget charges the priced cost. The worst case check before each call prices the cached prefix as a cache write. calls.jsonl records both cache counts per call, result.json records token totals and the cache hit rate, and the run summary, replay, compare output and live page show the hit rate.
The prefix tests now run over an explicit list of every prompt version, version 9 included, and a test fails if a new version is not added to it. The length check that keeps the cached prefix above the 512 token minimum for Claude Haiku 5.5 runs on versions 8 and 9.
AddisonTech
force-pushed
the
feature/prompt-caching
branch
from
October 24, 2026 20:34
283ab66 to
58b22e8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns on prompt caching for every model call and prices cached tokens correctly.
cache_controlephemeral breakpoint (default 5 minute TTL). It depends only on the envelope config and the building names, so it is byte-identical every hour; the hour's observation follows it in the user message.prompt_cache, and replay reads it (absent means off), so runs recorded before caching rebuild the exact request they sent and still replay.cache_read_input_tokensandcache_creation_input_tokensfrom every response. Cache writes are priced at 1.25x input and cache reads at 0.1x, in the model table. The run budget charges the priced cost, and its check before each call prices the cached prefix as a cache write.calls.jsonlrecords both cache counts per call,result.jsonrecords token totals and the cache hit rate, and the run summary, replay, compare output and live page show the hit rate.tests/test_prompt_cache.pyuse a fake client.The version 8 system prompt is about 870 tokens (estimated), above the 512 token minimum for Claude Haiku 5.5. Version 1 is about 400 tokens and will not cache; it was not padded.
Based on the recorded version 8 month ($0.418), the expected cost drop is about 13% if only the system prompt is cached, and up to about 41% if the API's structured output preamble sits inside the cached prefix. The first cached run's hit rate will show which.
Rebased onto main after PR #12 (release-0.11.0) merged. Prompt version 9 uses the same cached system block in
build_request, and the caching tests cover it explicitly. The version 9 system prompt is 3,886 characters, about 925 tokens (estimated), above the 512 token minimum.Tests: 320 passed, 34 skipped (CityLearn not installed locally).
scripts/check_repo.pypassed. All 29 recorded runs, including the 8 version 9 runs, and the committed baseline replay identically.