Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Reduce Dory's long-prompt time to first token.
2. What is the highest-leverage variable for that outcome?
The cost of running attention over long prompts during prefill.
3. How will this work move that variable rather than create busywork?
Use an opt-in Triton kernel that tiles queries and reuses KV reads. Keep per-query attention for batches containing a short prompt.
4. Who is the customer or user, and what do they actually need?
Dory users serving long prompts on RTX 5090 who need lower time to first token and clear hardware and BF16 reproducibility limits.
PR necessity and bugfix evidence
A PR is needed to expose and validate the packed full and sliding-window attention path for Dory. This is an opt-in performance feature.
Purpose
ExecutionConfig(dory_packed_prefill=True)and--dory-packed-prefill. The default is per-query attention.Hardware coverage
GPU validation covers one RTX 5090 only. Other SM 12.0 devices still need device-specific correctness and performance validation. Other GPU architectures require their own Triton tile and pipeline configuration before enabling this path. The 5090 kernel configuration stays within its per-block shared-memory limit for Dory's 256-dimensional heads.
BF16 output alignment
Test Plan
vllm-rlt Version / Commit
Base
d090670, headb17d1f6.pre-commit run --all-filesand full CPU regression suite.pytest -q --run-gpu tests/test_dory_gpu.py tests/test_attention.py -k 'not flash'. Cover full and sliding-window attention, skipping an entire key tile, compensated multiplication, and short-prompt routing in single-stream and default multi-stream execution.Test Result
Each mixed p95 is based on 18 short requests. Separate operator-only and full profiles of a warmed 256-token prefill step at 4K context observed 112 attention kernel launches with per-query attention and 56 with packed attention. The operator-only trace measured 92.7 ms and 9.25 ms of summed attention-kernel device time, respectively.