Skip to content

Commit 1ebb119

Browse files
Add Qwen3 14B A8W8 kernels (#642)
## Summary Add the Qwen3-14B A8W8 kernel implementation used by the native serving path. This PR keeps the existing BF16 Qwen3 path isolated and introduces separate A8W8 prefill/decode modules for the quantized model: - add `prefill_hidden` support for Qwen3-14B A8W8 hidden-state prefill chunks - add the optimized A8W8 `decode_fwd` layer kernel used by serving decode - handle INT8 weights, activation/weight scales, paged KV cache scale metadata, and A8W8-specific decode constants inside the A8W8 modules - keep scheduling, model loading, tokenizer, and request orchestration in `pypto-serving` rather than carrying standalone lib-side runners - keep the ordinary BF16 Qwen3-14B execution path separate from the A8W8 path ## Implementation Notes The lib-side deliverable is intentionally limited to kernels and kernel-adjacent compatibility: - `models/qwen3/14b/prefill_fwd_a8w8.py` implements the A8W8 prefill hidden path - `models/qwen3/14b/decode_layer_a8w8.py` implements the A8W8 decode layer path used by serving - `golden/runner.py` compatibility changes keep generated/runtime artifacts runnable for the JIT path - obsolete standalone debug/golden entry points were removed during slimming; debug-stage switches in the kernel remain available for targeted numerical diagnosis ## Validation End-to-end validation was run through the native serving stack with this kernel PR plus the matching serving/backend PRs: - prompt: `介绍一下北京故宫` - generated tokens: 48 - output quality: normal Chinese continuation - TTFT: `7.054s` - TPOT: `67.8 ms/token` - decode throughput: `14.76 tok/s` Focused serving and PyPTO checks also passed in the matching PRs. ## Related PRs / Issues - Tracking issue: #665 - Serving-side PR: hw-native-sys/pypto-serving#48 - PyPTO backend/lowering PR: hw-native-sys/pypto#1920 Co-authored-by: vegetabledoww <vegetabledoww@users.noreply.github.com>
1 parent 32ed0e7 commit 1ebb119

4 files changed

Lines changed: 1741 additions & 1 deletion

File tree

models/qwen3/14b/config.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@
3434

3535
# Numeric constants.
3636
EPS = 1e-6
37+
INT8_SCALE_MAX = 127.0
38+
INT8_AMAX_EPS = 1e-4
3739
HIDDEN_INV = 1.0 / HIDDEN
3840
HEAD_DIM_INV = 1.0 / HEAD_DIM
3941
ATTN_SCALE = 1.0 / (HEAD_DIM ** 0.5)

0 commit comments

Comments
 (0)