Commit 1ebb119
authored
Add Qwen3 14B A8W8 kernels (#642)
## Summary
Add the Qwen3-14B A8W8 kernel implementation used by the native serving
path.
This PR keeps the existing BF16 Qwen3 path isolated and introduces
separate A8W8 prefill/decode modules for the quantized model:
- add `prefill_hidden` support for Qwen3-14B A8W8 hidden-state prefill
chunks
- add the optimized A8W8 `decode_fwd` layer kernel used by serving
decode
- handle INT8 weights, activation/weight scales, paged KV cache scale
metadata, and A8W8-specific decode constants inside the A8W8 modules
- keep scheduling, model loading, tokenizer, and request orchestration
in `pypto-serving` rather than carrying standalone lib-side runners
- keep the ordinary BF16 Qwen3-14B execution path separate from the A8W8
path
## Implementation Notes
The lib-side deliverable is intentionally limited to kernels and
kernel-adjacent compatibility:
- `models/qwen3/14b/prefill_fwd_a8w8.py` implements the A8W8 prefill
hidden path
- `models/qwen3/14b/decode_layer_a8w8.py` implements the A8W8 decode
layer path used by serving
- `golden/runner.py` compatibility changes keep generated/runtime
artifacts runnable for the JIT path
- obsolete standalone debug/golden entry points were removed during
slimming; debug-stage switches in the kernel remain available for
targeted numerical diagnosis
## Validation
End-to-end validation was run through the native serving stack with this
kernel PR plus the matching serving/backend PRs:
- prompt: `介绍一下北京故宫`
- generated tokens: 48
- output quality: normal Chinese continuation
- TTFT: `7.054s`
- TPOT: `67.8 ms/token`
- decode throughput: `14.76 tok/s`
Focused serving and PyPTO checks also passed in the matching PRs.
## Related PRs / Issues
- Tracking issue: #665
- Serving-side PR: hw-native-sys/pypto-serving#48
- PyPTO backend/lowering PR: hw-native-sys/pypto#1920
Co-authored-by: vegetabledoww <vegetabledoww@users.noreply.github.com>1 parent 32ed0e7 commit 1ebb119
4 files changed
Lines changed: 1741 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
0 commit comments