5080deploy: generic fixes from 1080deploy (PLE per-fill readback events, host-tier probes, GDN workspace reserve, server fixes, --embed-weights host off) by gdevenyi · Pull Request #77 · gdevenyi/FreeToken

gdevenyi · 2026-10-02T05:51:57Z

Stacked on #76 (the upstream PRs). This brings in the generic fixes from the sibling branch 1080deploy (same model, GTX 1080 Ti) as cherry-pick -x commits, plus test fix-ups for this stack. Pascal-only tuning, CPU-MoE/NUMA, mxfp8 load-time quant and bench tooling are left out.

Disk PLE (the production decode path):

52a1801: skip the device n-gram hash for the disk table, which reads only its shape.
79988cc: one readback event per graph fill. A shared event re-recorded by the next step before the filler reached synchronize() made fill N wait on graph N, which waits on fill N. A fill that failed after the next step submitted was dropped silently.
198574d: recycle those events on the engine thread. Destroying them on the filler can block behind a blocked cuGraphLaunch.
New regression test test_ple_disk_no_graph_fence.py. It guards 5080deploy: rebuild on upstream main d3512b4 (upstream #546 max_tokens fix, #548 row store, #588 eager PLE buffers without its graph fence) #75's decision to drop fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588's _graph_consumed: it passes here and fails with TimeoutError: a fill waited on stream position 2 when the fence is restored.

GDN host tier / kvcache:

b349af0: exact host-tier probe for a tombstone prefix (scheduler CPU per tombstone ~1.8 s → ~4 ms at 262K keys).
44f2506: re-probe the host entry after making room, which could evict the hit.
eb462bd: refuse the host KV-page tier on a pool whose index slab it cannot restore.
6751cf8: keep one GPU GDN snapshot slot when the host tier is on. Inert here, because --mamba 8 is explicit.
af5ca25: test-only.

Engine:

ec8731a, 9d836ed: reserve the GDN prefill workspace in the cache budget and in the rebuild fit-check. The 1650-slot post-start rebuild is still accepted.
8e4891f: count min_tokens progress from the device length. Under overlap, EOS stayed masked one token past min_tokens.

Server:

8276ccd: protocol-level thinking requests win over --default-thinking-mode.
ab2693c: type allOf-wrapped refs and untyped enum/const tool parameters.
7ccc433: /v1/models/.
55a8e72: the Anthropic thinking block carries signature.
50f99d1: contract tests.
c6b7ce7: help text.
e60b2c3: keep developer ahead of system (Codex), and reject images merged into system.

MoE / kernels / tests:

37c2479: the small-prefill path, off here.
dbe82ad: the Triton sampling softmax was NaN over a fully masked prefix. That is the fallback when flashinfer is absent. Its test moved to tests/kernels/test_triton_sampling.py, which fails on the old code.
e700e8f, cedf179: chunked-vs-one-shot prefill tests.

--embed-weights host (9f2b7a2, ccfb2b8, cc5af7a, a8b2f0a, 579e03a, default gpu, off here). This keeps the 1.18 GiB token embedding in pinned host RAM, gathered over UVA. It lands inert; turning it on (and spending the VRAM on expert slots) is a separate A/B.

Fix-ups for this stack: test fakes for FlashML-org#573's hf_overrides argument and our ensure kwargs, import pytest, and n=0 (n=2 is served here).

Validation of the whole stack (RTX 5080, production config: 1650 slots, rule eviction, copy overlap, prefetch budget 4):

Unit tests:
- CPU suite: only the known test_mp_coalesced environment failure.
- GPU suite: the 4 known environment failures plus one prefetch timing test (see below).
- Targeted GPU tests: 231 passed.
- Negative controls on the old code: fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588's eager test and the softmax test both fail there.
Decode, interleaved A/B, 2 rounds, tok/s:

Build 32K 128K

old 51.23 / 51.27 49.73–49.98

new 51.24 / 51.25 49.74–49.90

At 255K (256 output tokens): old 48.92 / 48.95, new 48.90 / 48.86. Misses/token: 110.9 old vs 111.1 new.
Greedy:
- New at max_tokens=255 matches old at 256 (old delivered 255) on the 3 fixed prompts.
- Cold-nonce prompts: 0/60 mismatches.
- New cold first-run fingerprints at 256: code a94b98014cebf515, ctx 6fd71cf37dc18731, think e377cb534e260935.
- A prefix-cached rerun of the same prompts gives different ctx/think text on this model (a known prefix-cache effect), so fingerprints compare cold runs only.
Quality: 14/15, unchanged.
VRAM, 255K prompt + side calls: peak 15636 MiB, was 15716.
Soak: 60 min on the deployed build (agent main loop with tool calls plus side calls, overlap, wait-sync PLE): 241 main turns, 152 side calls, 1461 tool calls, no error, longest data gap 19 s.
Flaky test: one full GPU suite run failed test_moe_prefetch_on_install_may_evict_a_slot_whose_prefetch_copy_is_in_flight[rule-2-eager]. It did not reproduce in 8 isolated runs of all 8 params on each of the old and new trees, nor in a whole-file run (378 passed).
Fork issue detokenizer duplicates text when one call holds several messages for the same uid (latent; blocks multi-token emission) #69's repro (same-uid detokenize duplication) is fixed by fix(scheduler): match stop strings with incremental decoding FlashML-org/FreeToken#464's guard.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…s only its shape DiskRowTable hashes on the host in its fill; its lookup and prefetch read only the row_ids shape, so every prefill chunk and captured decode step ran the hash for nothing. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 52a1801)

…pending fill A shared event re-recorded by the next step before the filler reached synchronize() made fill N wait on graph N, which waits on fill N; and a fill that failed after the next step submitted was dropped, so its graph ran on stale rows without an error. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 79988cc)

… the filler The filler thread dropped each fill's last event reference, and cuEventDestroy there blocks on the driver lock that a blocked cuGraphLaunch holds (host-func nodes in the decode graph make the launch wait for the GPU, parked on the next fill's WAIT), so decode deadlocked with FREETOKEN_CPU_MOE_FLAG_SYNC=0. Events now return to an engine-thread pool once their fill is done. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 198574d)

… it cannot restore Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit eb462bd)

…tier is on Evicted snapshots spill to RAM and return on a hit, so the floor of four resident ones only costs VRAM: three 110 MiB Qwen3.8-Flash-Next slots go back to the expert cache. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 6751cf8)

Revival only accepts an exact entry, but the longest-prefix lookup re-hashed every shorter page-aligned prefix on a miss and refreshed the LRU age of a shorter entry it then rejected. _rehydrate_snapshot on a 262144-token key, CPU, median of 3: no host entry 1852 ms -> 4.0 ms entry 4096 tokens in 1494 ms -> 2.9 ms entry 8192 below 91 ms -> 3.5 ms exact hit 1.4 ms -> 1.6 ms Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit b349af0)

…e it The evictions that make room for a revived snapshot or a host prefix write their victims to the host tier, which can drop the hit's own entry and hand its GDN and KV slots to the victim, so the stale hit restored another prefix's state. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 44f2506)

…page copies Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit af5ca25)

chunk_gated_delta_rule_fwd_h materializes the per-chunk recurrent states for the whole extend at once -- [NT, H, V, K] with NT = ceil(extend/64) -- plus a transient v_new buffer. Subtract it at the startup sizing and the --moe-cache-auto plan for hybrid-GDN models. (cherry picked from commit 5c3b914) Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit ec8731a)

…ck too The cherry-picked reservation covered startup sizing and the auto plan but not validate_rebuild, so a runtime rebuild could grow the pools back into it. Also price the scheduler's default extend when the config is a bare EngineConfig, and document that h + v_new is a lower bound on the GDN prefill peak. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 9d836ed)

Under overlap scheduling the sampler prepares batch N before batch N-1's token lands in req.input_ids, so the host count lagged by one and kept the stop ids masked for one token past min_tokens. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 8e4891f)

Assisted-by: Claude Opus 5.5

…thinking-mode returned early, so reasoning_effort / reasoning.effort / thinking.type were silently turned off under --default-thinking-mode chat. count_tokens also skipped the default. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 8276ccd)

…meters FlashML-org#435 sends any parameter it cannot type to the loose JSON parser, so an untyped string enum member such as "1" came back as the integer 1, outside the enum. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit ab2693c)

Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 7ccc433)

The full response and the closing signature_delta already send signature="", but exclude_none dropped it from content_block_start, where the SDK types it as required. Taken from upstream FlashML-org#517 (b044276). Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 55a8e72)

Upstream FlashML-org#517 tests without its docs and pyproject dev-extra: the SDK tests importorskip anthropic/openai/httpx so the server suite still collects without them. (cherry picked from commit b044276, tests only) Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 50f99d1)

…lp, which it does not serve Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit c6b7ce7)

…rging system messages render_messages hoisted system ahead of a leading developer message, which the tokenizer then merged as "S\n\nD"; it now leaves developer requests to the tokenizer's per-template mapping. Both merge sites joined str() of an image part list into the prompt, data URL included; they now raise the template's 400 instead. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit e60b2c3)

…is served here Assisted-by: Claude Opus 5.5

… in every layer An unpinned layer's copy_missing is the whole-layer pageable copy, which cannot serve the LRU slot remap, so it keeps streaming. The per-layer fallback when a layer routes to more than half the cache is gone: the engine's num_experts slot floor already keeps every chunk resident, and a uniform path cannot leave a later layer in an overlap buffer that was never begun. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 37c2479)

… kwargs Assisted-by: Claude Opus 5.5

`torch.equal` fails for all three cut points on torch 2.11.0+cu130 / flashinfer 0.6.18 / triton 3.6.0 (sm_89): cuBLAS picks a different kernel for M=5000 and M=5000-cut, so the indexer's q and k round differently, a few rows flip one block at the top-k margin, and the layer output differs by up to 2.6e-3 (a few bf16 ulps). Both paths are deterministic run to run. Use the tolerance of the neighbouring oracle tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt (cherry picked from commit 3849e3c) Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit e700e8f)

…o the one-shot run At bf16 output tolerance a broken dual-source compress only swaps a block in a few rows and passes; the slab keys are unit-scale, so a wrong straddling-group key stands out. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit cedf179)

…ked prefix min_p (FlashML-org#393) writes -inf over most of the vocabulary; a column chunk or leading block that is -inf throughout computed exp(-inf - -inf) = NaN and poisoned the whole row. Pascal samples through this softmax (no flashinfer below sm_75), so min_p > 0 drew uniform noise. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit dbe82ad)

…ights host) The embedding is a row lookup, so a single-GPU engine can leave the table in page-locked host RAM and gather each step's rows over UVA with the PLE row kernel (5 KiB per decode token, graph-safe): 1.2 GiB of VRAM back for Qwen3.8-Flash-Next. Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 9f2b7a2)

…lacement Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit ccfb2b8)

No regression test: overflowing it needs an output of 2^31+ elements (4 GiB+ in bf16). Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit cc5af7a)

Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit a8b2f0a)

… 1 at config time Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 579e03a)

…lashML-org#573 passes Assisted-by: Claude Opus 5.5

…am so a post-launch fence fails as a hang Upstream FlashML-org#588's _graph_consumed fence is left out of the merge: our graph fill runs on the filler thread before its own launch passes its WAIT, so that fence deadlocks the second step. Assisted-by: Claude Opus 5.5

Copilot

Copilot review overview

🟡 Changes recommended

The GDN workspace estimate under-reserves varlen batches and over-reserves tensor-parallel heads.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Size h storage for worst-case concurrent partial sequences · New
Use TP-local value-head count for workspace sizing · New

What changed in this PR

This stacked PR ports deployment fixes across PLE, GDN caching, server compatibility, sampling, MoE, and optional host-resident embeddings.

Changes:

Fixes PLE fill synchronization, host-tier restoration, sampling, and cache accounting.
Improves protocol compatibility, thinking controls, tool schemas, and message handling.
Adds broad CPU/GPU regression and contract coverage.

File	Description
`python/freetoken/engine/config.py`	Adds host-embedding configuration and validation.
`python/freetoken/engine/engine.py`	Places embeddings on host and reserves GDN workspace.
`python/freetoken/engine/sample.py`	Uses device progress for `min_tokens`.
`python/freetoken/kernel/triton/ple.py`	Supports additional host-table dtypes and large offsets.
`python/freetoken/kernel/triton/sampling.py`	Avoids NaNs in masked softmax chunks.
`python/freetoken/kvcache/host_prefix_tier.py`	Adds exact lookup and QSA safeguards.
`python/freetoken/kvcache/linear_state_pool.py`	Adjusts snapshot sizing and GDN budgeting.
`python/freetoken/layers/embedding.py`	Implements host-resident embedding lookup.
`python/freetoken/layers/moe.py`	Expands small-prefill on-demand loading.
`python/freetoken/models/qwen4_exp/ple.py`	Skips unused device-side hashing.
`python/freetoken/models/qwen4_exp/ple_disk.py`	Adds per-fill readback event lifecycle.
`python/freetoken/scheduler/cache.py`	Makes host-tier rehydration exact and eviction-safe.
`python/freetoken/server/anthropic_api.py`	Fixes thinking defaults and signatures.
`python/freetoken/server/args.py`	Adds host-embedding CLI option and help updates.
`python/freetoken/server/function_call_parser.py`	Resolves wrapped and literal schemas.
`python/freetoken/server/generation.py`	Preserves developer/system ordering safely.
`python/freetoken/server/model_meta.py`	Centralizes default thinking-mode merging.
`python/freetoken/server/openai_api.py`	Adds model retrieval and trailing-slash handling.
`python/freetoken/server/responses_api.py`	Preserves explicit reasoning controls.
`python/freetoken/tokenizer/tokenize.py`	Safely merges system/developer contents.
`tests/engine/test_cache_budget.py`	Covers rebuild and host-pinning accounting.
`tests/engine/test_embed_weights.py`	Tests host-embedding constraints.
`tests/engine/test_logits_processors.py`	Covers overlap progress and masked sampling.
`tests/kernels/test_host_embedding.py`	Validates host embedding gathers and graphs.
`tests/kernels/test_triton_sampling.py`	Regresses masked-prefix softmax NaNs.
`tests/kvcache/test_host_tier_live_pool.py`	Verifies copy-stream ordering.
`tests/kvcache/test_host_tier_nvfp4_guard.py`	Tests QSA host-tier rejection.
`tests/kvcache/test_host_tier_rehydrate_exact.py`	Tests exact, eviction-safe restoration.
`tests/kvcache/test_linear_state_pool_alloc.py`	Tests GDN workspace estimates.
`tests/models/qwen4_exp/test_ple.py`	Tests host-hashed table behavior.
`tests/models/qwen4_exp/test_ple_disk_fill_order.py`	Tests fill ordering and event recycling.
`tests/models/qwen4_exp/test_ple_disk_no_graph_fence.py`	Regresses graph-fill deadlocks.
`tests/models/qwen4_exp/test_qsa_backend.py`	Updates chunked-prefill equivalence checks.
`tests/moe/test_small_prefill_on_demand.py`	Covers expanded small-prefill routing.
`tests/scheduler/test_hybrid_cache_manager.py`	Tests host-tier snapshot floor.
`tests/server/test_accounting.py`	Adds accounting drain concurrency contracts.
`tests/server/test_anthropic_api.py`	Validates streamed thinking signatures.
`tests/server/test_default_thinking_mode.py`	Covers protocol-level thinking precedence.
`tests/server/test_function_call_parser.py`	Adds detector contract coverage.
`tests/server/test_function_call_parser_schema_resolution.py`	Tests wrapped refs and literal types.
`tests/server/test_mm_input.py`	Tests developer ordering and image rejection.
`tests/server/test_openai_api.py`	Tests model routes.
`tests/server/test_protocol_compat.py`	Adds SDK-level protocol contracts.
`tests/server/test_reasoning_parser.py`	Adds reasoning-parser contracts.
`tests/server/test_supervisor.py`	Tests shutdown-aware worker failure handling.
`tests/tokenizer/test_tokenize.py`	Tests unsafe system-image merging.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        return 0
+    # EngineConfig has no max_extend_tokens (the served config is a SchedulerConfig subclass)
+    extent = min(getattr(config, "max_extend_tokens", 8192), config.max_seq_len)
+    nt = -(-extent // _GDN_CHUNK_SIZE)


+    states = (
+        nt
+        * linear_group.num_value_heads
+        * linear_group.value_head_dim
+        * linear_group.key_head_dim
+    )
+    v_new = extent * linear_group.num_value_heads * linear_group.value_head_dim


gdevenyi · 2026-10-02T07:30:47Z

--embed-weights host is now on in production, and the freed VRAM went to expert slots (1650 → 2100). Measured on the deployed stack with #79, interleaved over 2 rounds:

arm	idle VRAM	decode 32K	decode 128K	misses/token
base, 1650 slots	14562 MiB	50.29 / 50.60	48.72 / 49.84	113.3
host embedding, 1650 slots	13384 MiB	50.30 / 50.53	48.70 / 49.81	113.3
host embedding, 2100 slots	14564 MiB	52.57 / 52.90	50.77 / 52.13	96.0

The host gather costs nothing at decode and frees 1178 MiB. Spending it on 450 slots gives +4.3–4.5% decode.
Greedy fingerprints are identical in every arm.
255K prompt with a side call mid-prefill and 3 after: peak 15496 MiB at 2100 slots vs 15514 MiB at 1650 (usable ~15882).
Host RAM: the server cgroup is 72.6 GiB (was ~71.2).
Final config: 30-min soak passed (135 main turns, 81 side calls, 1073 tool calls, no error); quality 14/15.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

gdevenyi and others added 30 commits October 1, 2026 23:21

fix(kvcache): refuse the host kv-page tier on a pool whose index slab…

81c6137

… it cannot restore Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit eb462bd)

test(kvcache): cover the engine-stream waits of the prefix tier's kv …

ebaa599

…page copies Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit af5ca25)

test(engine): import pytest for the parametrized min_tokens test

219cc98

Assisted-by: Claude Opus 5.5

fix(server): keep /v1/models/ listing the model after the retrieve route

86674ac

Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 7ccc433)

fix(server): drop /v1/completions from the --enable-metrics-report he…

e2c1de4

…lp, which it does not serve Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit c6b7ce7)

test(server): ask for n=0 in the invalid-request sdk test, since n=2 …

21261f6

…is served here Assisted-by: Claude Opus 5.5

test(moe): let the small-prefill ensure fake take the eviction policy…

9be869b

… kwargs Assisted-by: Claude Opus 5.5

fix(layers): gather fp32 host embeddings and refuse other dtypes at p…

d4d9e52

…lacement Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit ccfb2b8)

fix(kernel): compute the host gather output offset in int64

243c70b

No regression test: overflowing it needs an output of 2^31+ elements (4 GiB+ in bf16). Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit cc5af7a)

fix(engine): charge the host embedding's pinned bytes to the pin budget

1140b36

Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit a8b2f0a)

fix(engine): refuse --embed-weights host for a tied embedding or tp >…

764c357

… 1 at config time Assisted-by: Claude Opus 5.5 (1M context) (cherry picked from commit 579e03a)

gdevenyi added 2 commits October 1, 2026 23:26

test(engine): fake cached_load_hf_config with the overrides argument F…

99f01b2

…lashML-org#573 passes Assisted-by: Claude Opus 5.5

Copilot AI balanced review requested due to automatic review settings October 2, 2026 05:51

Copilot started reviewing on behalf of gdevenyi October 2, 2026 05:52

View session

gdevenyi mentioned this pull request Oct 2, 2026

5080deploy 9/9: merge upstream #389 (+#385), load-time FP8 dense projections #17

Open

Copilot AI reviewed Oct 2, 2026

View reviewed changes

gdevenyi mentioned this pull request Oct 2, 2026

5080deploy: merge upstream #578, NVFP4 prefill dequant by bit placement (-32% TTFT at 64K, bit-identical) #79

Open

Navigation Menu

Repository navigation

5080deploy: generic fixes from 1080deploy (PLE per-fill readback events, host-tier probes, GDN workspace reserve, server fixes, --embed-weights host off) - #77

5080deploy: generic fixes from 1080deploy (PLE per-fill readback events, host-tier probes, GDN workspace reserve, server fixes, --embed-weights host off)#77
gdevenyi wants to merge 32 commits into
5080/34-upstream-prsfrom
5080/35-1080deploy-fixes

gdevenyi commented Oct 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

gdevenyi commented Oct 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Build	32K	128K
old	51.23 / 51.27	49.73–49.98
new	51.24 / 51.25	49.74–49.90

Conversation

gdevenyi commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

gdevenyi commented Oct 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gdevenyi commented Oct 2, 2026 •

edited

Loading