Skip to content

[Engine] native_dense: STRATA_HC_Q8 uploads only the stage's own Q8_0 rows on a trimmed layer split - #3559

Open
havok2-htwo wants to merge 1 commit into
Niko1221:mainfrom
havok2-htwo:hcq8-020-beta
Open

havok2-htwo wants to merge 1 commit into
Niko1221:mainfrom
havok2-htwo:hcq8-020-beta

Conversation

@havok2-htwo

Copy link
Copy Markdown
Contributor

[Engine] native_dense: STRATA_HC_Q8 uploads only the stage's own Q8_0 rows on a trimmed layer split

Problem

With --trim-stage-weights and STRATA_HC_Q8=1, the loop that attaches the GGUF's Q8_0 hyper-connection rows does not skip other stages' layers. Each trimmed card uploads all 48 layers' rows and reads only its own. Observed: 0.63 GiB per card with UD-IQ4_XS.

Change

src/core/native_dense.cpp, +2 lines: the loop skips a tensor when outside(name) || !in_range(name), the filter the upload loop above it already uses. No value changes; without stage trim in_range passes every layer, so nothing changes.

Validation

  • Revision: v0.2.0-beta 3d1fb480 plus this commit (023b3962), CUDA sm_86 build with setup-style flags, 4x RTX 3090 24 GB.
  • Rig: Qwen3.8-Flash-Next UD-IQ4_XS native GGUF, --layer-split 12,25,37 --trim-stage-weights, STRATA_HC_Q8=1, 16 slots, greedy.
  • Per-card STRATA_HC_Q8=1: ... GiB line: stock 0.63 / 0.63 / 0.63 / 0.63; with this change 0.16 / 0.18 / 0.16 / 0.15.
  • Tokens (tools/batch_test.py --batch 16 --n 16 --max-new 128, same expert cache on every card): solo 16/16 and batch 16/16 identical between stock and this change.
  • Without trim, 0.63 GiB per card with and without the change, as expected.

Not tested

HIP, Metal and SYCL builds (this file is compiled there, but not built here), 2- and 3-GPU splits, --layer-split auto, long prompts, other models.

…a trimmed layer split

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
bsorensen110 added a commit to bsorensen110/Strata that referenced this pull request Oct 22, 2026
…e's own Q8_0 rows (trimmed split)) — integ batch 19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant