Skip to content

perf(dory): add opt-in packed Triton prefill - #189

Open
cecilsd wants to merge 3 commits into
InferMatrixLab:mainfrom
cecilsd:perf/dory-packed-prefill
Open

cecilsd wants to merge 3 commits into
InferMatrixLab:mainfrom
cecilsd:perf/dory-packed-prefill

Conversation

@cecilsd

@cecilsd cecilsd commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Required human contributor answers

1. What outcome are you trying to achieve?

Reduce Dory's long-prompt time to first token.

2. What is the highest-leverage variable for that outcome?

The cost of running attention over long prompts during prefill.

3. How will this work move that variable rather than create busywork?

Use an opt-in Triton kernel that tiles queries and reuses KV reads. Keep per-query attention for batches containing a short prompt.

4. Who is the customer or user, and what do they actually need?

Dory users serving long prompts on RTX 5090 who need lower time to first token and clear hardware and BF16 reproducibility limits.

PR necessity and bugfix evidence

A PR is needed to expose and validate the packed full and sliding-window attention path for Dory. This is an opt-in performance feature.

Purpose

  • Add ExecutionConfig(dory_packed_prefill=True) and --dory-packed-prefill. The default is per-query attention.
  • Use 32 × 64 tiles, one pipeline stage, and a residual term for BF16 rounding of the softmax weights. The option requires Dory BF16, NVIDIA SM 12.0, per-loop KV, and ordinary synchronous eager prefill.
  • Use packed attention in PRELUDE, the recurrent loops, and prompt-history CODA when every prompt in the batch is longer than 64 tokens. Any prompt of 64 tokens or less keeps the batch on per-query attention. First-token sampling and decode use per-query attention.

Hardware coverage

GPU validation covers one RTX 5090 only. Other SM 12.0 devices still need device-specific correctness and performance validation. Other GPU architectures require their own Triton tile and pipeline configuration before enabling this path. The 5090 kernel configuration stays within its per-block shared-memory limit for Dory's 256-dimensional heads.

BF16 output alignment

  • Same batch shapes and token histories. Both paths use the same causal and sliding-window ranges. The packed kernel changes the floating-point reduction order and rounds intermediate softmax weights for the matrix product with V. The compensation reduces this rounding error. Residual differences can propagate through Dory's recurrent layers. The 8K and 16K controlled runs matched all 144 short-request logit rows exactly with the per-query fallback. Long-request logits can still differ.
  • Natural request arrivals. A faster long prefill can change which requests share a batch, affecting BF16 even on the default path. A controlled 16K replay using only per-query attention reproduced the same two short-request output differences when the admission schedule changed. The timed runs matched 12/12 long-request sequences and 34/36 short-request sequences. For strict reproduction, use per-query attention and fix the batch shapes.

Test Plan

vllm-rlt Version / Commit
Base d090670, head b17d1f6.

  • Linux pre-commit run --all-files and full CPU regression suite.
  • RTX 5090 pytest -q --run-gpu tests/test_dory_gpu.py tests/test_attention.py -k 'not flash'. Cover full and sliding-window attention, skipping an entire key tile, compensated multiplication, and short-prompt routing in single-stream and default multi-stream execution.
  • Real Dory 4B BF16 checkpoint on one RTX 5090, PyTorch 2.13.0+cu130, Triton 3.7.1. Compare the same engine with the option off and on. Run three interleaved pairs after warmup. Use two loops, 256-token prefill chunks, and fixed output lengths of 8 long-request tokens and 12 short-request tokens. The mixed load adds six 32-token requests while the long prompt is in prefill.

Test Result

  • All Linux pre-commit hooks passed. The standalone full Linux CPU suite recorded 1,047 passed, 744 skipped. The targeted GPU suite recorded 132 passed on RTX 5090.
  • Windows pre-commit lint and formatting passed. Two unrelated Windows CPU failures, in Huginn dense-reference parity and PD worker-exit handling, also reproduced on the base commit.
Workload and metric Per query Packed with short-prompt fallback
8K single-request median TTFT 4.28 s 1.85 s
16K single-request median TTFT 12.02 s 3.86 s
8K mixed median batch completion 5.91 s 3.70 s
16K mixed median batch completion 13.54 s 5.65 s
8K mixed short-request TTFT p95 284 ms 274 ms
16K mixed short-request TTFT p95 284 ms 248 ms

Each mixed p95 is based on 18 short requests. Separate operator-only and full profiles of a warmed 256-token prefill step at 4K context observed 112 attention kernel launches with per-query attention and 56 with packed attention. The operator-only trace measured 92.7 ms and 9.25 ms of summed attention-kernel device time, respectively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant