Skip to content

[Dory] Enable per-loop prefix caching - #190

Open
HowardYangAI wants to merge 2 commits into
InferMatrixLab:mainfrom
HowardYangAI:dory-2c-prefix
Open

HowardYangAI wants to merge 2 commits into
InferMatrixLab:mainfrom
HowardYangAI:dory-2c-prefix

Conversation

@HowardYangAI

@HowardYangAI HowardYangAI commented Nov 16, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?

Avoid repeating prefill computation for Dory requests with shared prefixes (scope).

2. What is the highest-leverage variable for that outcome?

The number of shared prompt tokens that are recomputed.

3. How will this work move that variable rather than create busywork?

Publish the KV each plane owns and restore it on a hit, so those prompt tokens can be skipped (reported failure and approach).

4. Who is the customer or user, and what do they actually need?

Dory users sending requests with shared prompt prefixes need to avoid repeating that work.

PR necessity and bugfix evidence

A PR is needed: Dory rejects prefix caching, and removing that guard still leaves publication blocked because boundary layers only write plane 0 (failure report); the publication failure was also reproduced locally.

Purpose

Enable prefix caching in per_loop mode: cache boundary KV at plane 0 and recurrent KV at every owned plane, then recompute the prompt tail to recover the final hidden state.

The branch includes Dragonborn's 2B prerequisite as the separate commit 4ce2fef; 0837c45 contains our prefix work and integration fixes, so the diff against main includes both.

Prefix caching remains disabled in last_loop mode and when SWA reclamation is enabled; graph, async and speculative execution use the existing all-layer layout.

Addresses #176; Dory packed prefill in #189 needs separate integration with grouped KV.

Test Plan

  • pre-commit run --all-files (Ruff lint, formatting and CPU regressions).
  • Cold/hit/branch parity at 1/2/4 loops with grouped and all-layer KV, plus eviction, sharing, event ordering and cleanup.

vllm-rlt Version / Commit: main c338aaa, prerequisite 4ce2fef, head 0837c45.

Test Result

All results below correspond to the current PR head 0837c45; the GPU tests were rerun after the fixes and rebases.

Check Result
Pre-commit: Ruff lint, format and CPU regressions Passed
Standalone full CPU suite 1120 passed, 753 skipped
Dory GPU regressions plus upstream state-pool/async regressions 117 passed, no skips
Real-checkpoint oracle, backend parity and generation stopping 4 passed, no skips
Real-checkpoint prefix reuse 20 configurations, 60/60 cold/hit/branch output comparisons matched

GPU environment: one RTX PRO 5000 72GB (SM120), Python 3.12.3, PyTorch 2.14.0+cu130, Triton 3.8.0; checkpoint from the ModelScope mirror, verified against its file hashes (weights SHA-256: 207483664dfef54c4c4438863260d3ba423bf3d6009b43fe84264b281e9fda23).

Real-checkpoint results

The prompt had 1100 tokens of repeated text; the branch changed the last 44 token IDs, and each request generated six greedy tokens with 256-token prefill chunks and 16-token cache blocks.

Coverage Configurations Cold/hit/branch output comparisons
FP32, 1/2/4 loops, four execution modes 12 36/36 matched
FP32, 2/4 loops, block and pipeline speculation 4 12/12 matched
BF16, 2 loops, four execution modes 4 12/12 matched

Every configuration reused 1088 tokens for the repeat and 1056 tokens for the branch, and released all request-owned KV pages.

BF16, two loops; time to the first token returned by the engine, in seconds:

Mode Cold: one request Repeat: paired with branch Branch: paired with repeat
Sync eager 0.776 0.110 0.110
Sync Graph 0.626 0.265 0.265
Async eager 0.460 0.102 0.102
Async Graph 0.983 0.647 0.647

These are single validation runs: the cold request ran first, then the repeat and branch ran together in the same engine; Graph timings may include capture, so the table does not establish a speedup. Outputs were checked against uncached sync-eager runs with the same dtype and loop count. The repeated-text inputs test cache reuse, not language-model quality.

BF16 can round differently when chunk shapes change; these comparisons cover only the tested inputs. FlashAttention-4 remains unsupported on SM120: the six generic FA4 cases reported earlier were rejected by the architecture guard, and were not part of this Triton rerun.

prettygirlisnotme and others added 2 commits November 16, 2026 16:04
Publish complete prefix pages using boundary and recurrent plane ownership,
retain cached pages after request release, and restore coverage on hits.
Use incremental prefix hashing and completion-aware publication, releasing
pending records with their allocation.

Keep compatible all-layer storage for graph, async and speculative execution,
and harden grouped allocation, padding, reclamation and initialization paths.
Preserve CacheConfig positional compatibility and document CPU/GPU validation.

This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B
storage/reclamation change remains a separate commit by its original author.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants