diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 0fddc11e2e..da7734afec 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -1000,7 +1000,7 @@ bits of the answer: a switch that changes rounding can change a greedy answer at | Setting | Applies to | Off switch | What it does | | --- | --- | --- | --- | | `STRATA_PREFILL_CPU_SHARE` (`auto`, cap `STRATA_PREFILL_CPU_SHARE_MAX=1024`) | CUDA, one GPU, no `--batch`, prompt chunks under 1,024 tokens | `STRATA_PREFILL_CPU_SHARE=0` | The CPU pool computes the experts few of the chunk's tokens route to, while the GPU works on the rest. `auto` measures per layer how long each side takes and checks that sharing pays. Skipped with one log line when RAM or commit is short. Output changes in the last bits. Layer split, batch slots and HIP keep it off unless you set it. | -| `STRATA_ROUTE_TAIL_SKIP` (7) | CUDA, and HIP gfx1200/gfx1201; one GPU or a layer split | `STRATA_ROUTE_TAIL_SKIP=0` | A missed expert that every token of a verify window routes at rank 7 or lower is neither copied nor computed. Not applied when every expert is in VRAM, with `--batch` slots, with a peer tier or helper GPU, on other HIP and SYCL, or when 90% or more of the experts are cached (`STRATA_TAIL_SKIP_MAX_CACHED=` moves that line; it is 60% on an AMD layer split). Answers differ slightly from a run without it. | +| `STRATA_ROUTE_TAIL_SKIP` (7) | CUDA, and HIP gfx1200/gfx1201; one GPU or a layer split, including while a helper GPU is attached | `STRATA_ROUTE_TAIL_SKIP=0` | A missed expert that every token of a verify window routes at rank 7 or lower is neither copied nor computed. Not applied when every expert is in VRAM, with `--batch` slots, with a peer tier, on other HIP and SYCL, or when 90% or more of the experts are cached (`STRATA_TAIL_SKIP_MAX_CACHED=` moves that line; it is 60% on an AMD layer split). While a helper GPU is attached the skip narrows to file-tier misses: an expert the helper holds, or one already in the host copy (a resident file or the anonymous arena), is still computed; the helper's start-up line says so. Answers differ slightly from a run without it. | | PCIe share from the CPU's speed | CUDA, one GPU, no `--pcie-frac` | `--pcie-frac N` always wins; `STRATA_PCIE_FRAC_DEFAULT=old` keeps the probe-only rule | The share starts at the probe's value and is refined in the first decode windows from the CPU pool's time per missed expert: `L / (L + 1.85 R)`, rounded to 0.05 and kept between 0.10 and 0.60, changed at most three times. Each change prints a `PCIe share:` line. On HIP it is the default for gfx1200/gfx1201; `STRATA_PCIE_FRAC_DEFAULT=auto` turns it on for other HIP parts. `STRATA_PCIE_AUTO_SPLIT=1` (HIP, opt-in) refines a two-stage layer split. | | Chunk follows the host's memory | Linux file tier | `STRATA_PREFILL_HOST_PLAN=0` | See [Linux file tier](#linux-file-tier). | | Fused int8 prompt experts | gfx12 (R9700): IQ3_S and IQ3_XXS packs. 12.0 NVIDIA cards with 80 or more SMs: IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS gate/up | `STRATA_PF_FUSED=0` | Replaces the MMQ prompt kernels. `STRATA_PF_FUSED=1` opts in every covered format on other cards (on HIP gfx11 and gfx1200/gfx1201 too; see [AMD_HIP.md](AMD_HIP.md)). | diff --git a/docs/QUANTS.md b/docs/QUANTS.md index 1e07626bb1..f90e82eada 100644 --- a/docs/QUANTS.md +++ b/docs/QUANTS.md @@ -159,7 +159,8 @@ story 39.5 -> 53.3 tok/s, code 60.1 -> 80.4, after a 6K document 49.0 -> 68.8, 5 routes at rank 7 or lower is skipped) costs these files more than the 2-3-bit packs: UD-Q4_K_XL's yardstick KL against Q8_0 goes from 0.056 to 0.086 with every expert in RAM (R9700, the auto cache) and from 0.053 to 0.115 in the RAM budget mode (A4000, 1,500 GPU slots), for +4-10% decode in RAM and +26-67% in the budget mode (5 pairs each). -`STRATA_ROUTE_TAIL_SKIP=0` in a config's `"env"` turns it off. +`STRATA_ROUTE_TAIL_SKIP=0` in a config's `"env"` turns it off. These runs had no helper GPU; with one attached only file-tier misses are +skipped - an expert the helper holds, or one already in the host copy, is still computed. **The hyper-connections on NVIDIA:** the pack's `--compat-bf16` copy of the Q8_0 HC projections rounds them to BF16 (and doubles their bytes). `STRATA_HC_Q8=1` reads the file's own Q8_0 instead (on AMD gfx12 it is the default): on the diff --git a/include/strata/core/expert_source.hpp b/include/strata/core/expert_source.hpp index 45ac7f7427..89c8ae3b86 100644 --- a/include/strata/core/expert_source.hpp +++ b/include/strata/core/expert_source.hpp @@ -216,6 +216,9 @@ class ExpertSource { /// Plan v0.3 P5: whether `blob(layer, expert)` lies in page-locked, CUDA-registered memory, so an /// asynchronous host-to-device copy can DMA it directly (no staging copy on the CPU). virtual bool pinned(int64_t layer, int64_t expert) const { (void) layer; (void) expert; return false; } + /// Bytes already in a host copy (a resident complement, or an arena that is not a demand-paged + /// file mapping). A file or mmap miss is false. The tail skip asks only while a helper is attached. + virtual bool host_copy(int64_t layer, int64_t expert) const { (void) layer; (void) expert; return false; } /// Called once before the first expert of a layer. A source that reads from disk wants to start the read /// here so it overlaps the quantisation, and a prefetching source in Phase 3 wants the ids. @@ -916,6 +919,7 @@ class FileExpertSource : public ExpertSource { /// VRAM. Never touches the resident complement or staging buffers. Disabled by default. uint64_t release(int64_t layer, int64_t expert) override; bool pinned(int64_t layer, int64_t expert) const override; + bool host_copy(int64_t layer, int64_t expert) const override { return has_resident(layer, expert); } const uint8_t* device_alias(int64_t layer, int64_t expert) const override; bool pcie_layer(int64_t layer) const override; bool transient(int64_t layer, int64_t expert) const override; @@ -1282,6 +1286,7 @@ class ArenaExpertSource : public ExpertSource { const uint8_t* blob_at(int64_t layer, int64_t expert) const; int64_t reads() const { return reads_.load(std::memory_order_relaxed); } bool pinned(int64_t layer, int64_t expert) const override; + bool host_copy(int64_t layer, int64_t expert) const override; const uint8_t* device_alias(int64_t layer, int64_t expert) const override; /// Prefix pinning: the old rule (device_alias(layer, 0)); pin by need: at least one pinned range in the layer. bool pcie_layer(int64_t layer) const override; diff --git a/src/core/expert_source.cpp b/src/core/expert_source.cpp index 583ee07f58..7c63e7d30c 100644 --- a/src/core/expert_source.cpp +++ b/src/core/expert_source.cpp @@ -4910,7 +4910,10 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 const int tail_rank = tail_skip_rank(); bool skip_d[kMaxWindowEntries]; for (int q = 0; q < nd; ++q) skip_d[q] = false; - if (tail_rank > 0 && d.peer == nullptr && d.remote_count == 0) { + // No helper: skip a missed tail even when the bytes are in a host copy. + // A helper skips a file-tier miss only. A row it holds, or one whose bytes + // are already in the host copy, is still computed. + if (tail_rank > 0 && d.peer == nullptr) { int minr[kMaxWindowEntries]; for (int q = 0; q < nd; ++q) minr[q] = 99; for (int64_t i = 0; i < n; ++i) { @@ -4922,6 +4925,8 @@ void expert_pool_dispatch_multi(ExpertDispatch& d, const float* x_f, const int32 const int32_t e = ids[distinct[q]]; if (e < 0 || e >= d.n_expert || minr[q] < tail_rank) continue; if (d.host_res[(size_t) d.layers * (size_t) d.n_expert + (size_t) e] >= 0) continue; // resident + if (d.remote_count > 0 && + (helper_holds(e) || (d.src != nullptr && d.src->host_copy(d.layers, e)))) continue; skip_d[q] = true; --nmiss; ++d.tail_experts; @@ -6183,6 +6188,12 @@ bool ArenaExpertSource::pcie_layer(int64_t layer) const { return layer >= 0 && (size_t) layer < layer_pcie_.size() && layer_pcie_[(size_t) layer] != 0; } +bool ArenaExpertSource::host_copy(int64_t layer, int64_t expert) const { + if (map_ != nullptr || base_ == nullptr || layer < 0 || expert < 0 || expert >= n_expert_) return false; + const int64_t idx = layer * n_expert_ + expert; + return idx >= 0 && idx < blobs_; +} + bool ArenaExpertSource::pinned(int64_t layer, int64_t expert) const { if (base_ == nullptr || layer < 0 || expert < 0 || expert >= n_expert_) return false; const auto& lay = strata::kernels::cpu::expert_layout(); diff --git a/src/program/generate.cpp b/src/program/generate.cpp index 956d2c5625..9483006fda 100644 --- a/src/program/generate.cpp +++ b/src/program/generate.cpp @@ -11066,7 +11066,9 @@ int main(int argc, char** argv) { // a verify window routes at rank 7 or lower is neither copied nor computed (answers differ slightly from 0.1.41, // KL ~0.003-0.006; docs/DETAILS.md). =0 restores 0.1.41's bytes. Nothing to do, so no default and no line, when every // expert is in VRAM; off with --batch slots (a window then mixes requests, and one request's skip would depend on - // another's tokens) and, as in the dispatch, with a peer or helper GPU. A layer split is fine: each stage skips by itself. + // another's tokens) and, as in the dispatch, with a peer GPU. With no helper, a missed tail is skipped + // even when its bytes are in a host copy. With a helper, a file-tier miss is skipped; a row the helper + // holds, or one already in the host copy, is still computed. A layer split is fine: each stage skips by itself. { const char* tse = std::getenv("STRATA_ROUTE_TAIL_SKIP"); // #1884: with nearly every expert in the GPU cache (V100 32 GB, Coder IQ1_M: 94.8% of the experts, 99.9% decode hits) the few @@ -11104,7 +11106,12 @@ int main(int argc, char** argv) { "few are missed and skipping them costs more draft acceptance than it saves; " "STRATA_ROUTE_TAIL_SKIP=7 turns it on\n", 100.0 * (double) cached_experts / (double) host_res.size()); if (tail_default_ok) strata::core::set_tail_skip_rank(7); - if (strata::core::tail_skip_rank() > 0 && !all_experts_resident) + if (strata::core::tail_skip_rank() > 0 && !all_experts_resident && drive.d.remote_count > 0) + std::fprintf(stderr, "strata serve: STRATA_ROUTE_TAIL_SKIP=%d is ON%s: a file-tier miss that every token of a verify window routes at rank %d or " + "lower is skipped. An expert a helper holds, or one already in the host copy, is still computed; " + "STRATA_ROUTE_TAIL_SKIP=0 turns it off\n", strata::core::tail_skip_rank(), + tse == nullptr ? " (default)" : "", strata::core::tail_skip_rank()); + else if (strata::core::tail_skip_rank() > 0 && !all_experts_resident) std::fprintf(stderr, "strata serve: STRATA_ROUTE_TAIL_SKIP=%d is ON%s: a missed expert that every token of a verify window routes at rank %d or " "lower is skipped (+10..20%% decode on 12-16 GB cards that miss experts, answers differ slightly from 0.1.41); " "STRATA_ROUTE_TAIL_SKIP=0 turns it off\n", strata::core::tail_skip_rank(),