Repository navigation
[Dory] Enable per-loop prefix caching - #190
Open
HowardYangAI wants to merge 2 commits into
Open
HowardYangAI wants to merge 2 commits into
HowardYangAI wants to merge 2 commits into
Conversation
Publish complete prefix pages using boundary and recurrent plane ownership, retain cached pages after request release, and restore coverage on hits. Use incremental prefix hashing and completion-aware publication, releasing pending records with their allocation. Keep compatible all-layer storage for graph, async and speculative execution, and harden grouped allocation, padding, reclamation and initialization paths. Preserve CacheConfig positional compatibility and document CPU/GPU validation. This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B storage/reclamation change remains a separate commit by its original author.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Avoid repeating prefill computation for Dory requests with shared prefixes (scope).
2. What is the highest-leverage variable for that outcome?
The number of shared prompt tokens that are recomputed.
3. How will this work move that variable rather than create busywork?
Publish the KV each plane owns and restore it on a hit, so those prompt tokens can be skipped (reported failure and approach).
4. Who is the customer or user, and what do they actually need?
Dory users sending requests with shared prompt prefixes need to avoid repeating that work.
PR necessity and bugfix evidence
A PR is needed: Dory rejects prefix caching, and removing that guard still leaves publication blocked because boundary layers only write plane 0 (failure report); the publication failure was also reproduced locally.
Purpose
Enable prefix caching in
per_loopmode: cache boundary KV at plane 0 and recurrent KV at every owned plane, then recompute the prompt tail to recover the final hidden state.The branch includes Dragonborn's 2B prerequisite as the separate commit
4ce2fef;0837c45contains our prefix work and integration fixes, so the diff against main includes both.Prefix caching remains disabled in
last_loopmode and when SWA reclamation is enabled; graph, async and speculative execution use the existing all-layer layout.Addresses #176; Dory packed prefill in #189 needs separate integration with grouped KV.
Test Plan
pre-commit run --all-files(Ruff lint, formatting and CPU regressions).vllm-rlt Version / Commit: main
c338aaa, prerequisite4ce2fef, head0837c45.Test Result
All results below correspond to the current PR head
0837c45; the GPU tests were rerun after the fixes and rebases.GPU environment: one RTX PRO 5000 72GB (SM120), Python 3.12.3, PyTorch 2.14.0+cu130, Triton 3.8.0; checkpoint from the ModelScope mirror, verified against its file hashes (weights SHA-256:
207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23).Real-checkpoint results
The prompt had 1100 tokens of repeated text; the branch changed the last 44 token IDs, and each request generated six greedy tokens with 256-token prefill chunks and 16-token cache blocks.
Every configuration reused 1088 tokens for the repeat and 1056 tokens for the branch, and released all request-owned KV pages.
BF16, two loops; time to the first token returned by the engine, in seconds:
These are single validation runs: the cold request ran first, then the repeat and branch ran together in the same engine; Graph timings may include capture, so the table does not establish a speedup. Outputs were checked against uncached sync-eager runs with the same dtype and loop count. The repeated-text inputs test cache reuse, not language-model quality.
BF16 can round differently when chunk shapes change; these comparisons cover only the tested inputs. FlashAttention-4 remains unsupported on SM120: the six generic FA4 cases reported earlier were rejected by the architecture guard, and were not part of this Triton rerun.