5080deploy: rebuild on upstream main d3512b4 (upstream #546 max_tokens fix, #548 row store, #588 eager PLE buffers without its graph fence) by gdevenyi · Pull Request #75 · gdevenyi/FreeToken

gdevenyi · 2026-10-02T05:51:47Z

Stacked on #74. The base 5080/base-8ba20f6 is the previous 5080deploy tip. This PR rebuilds 5080deploy on upstream main d3512b4, which has 7 new commits:

cee23bf fix(runtime): correct overlap token limits and cache safety FlashML-org/FreeToken#546: overlap token limits and cache safety
f5b9700 refactor(ple): extract a shared disk row store FlashML-org/FreeToken#548: shared disk row store
0d652e7 feat(rocm): add RDNA3 and RDNA4 runtime foundation FlashML-org/FreeToken#132: ROCm foundation
5031e3d perf(qsa): reduce block top-k candidates in levels past 16 splits FlashML-org/FreeToken#571: QSA top-k levels
eba7120 feat(install): record the install root as FREETOKEN_HOME in environment.d FlashML-org/FreeToken#576: FREETOKEN_HOME
235e201 feat(rotary): size the rope table for YaRN and add --hf-overrides FlashML-org/FreeToken#573: YaRN rope table and --hf-overrides
d3512b4 fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588: fence PLE disk staging

How. All 52 first-parent commits of 5080deploy since 71eb1f0 were replayed onto d3512b4. Each upstream PR is re-merged with its original message, and each plain commit is cherry-picked. rerere supplied the old resolutions.

New resolutions:

tests/kernels/test_e4m3_compat.py (at feat(kvcache): add nvfp4 kv quantization FlashML-org/FreeToken#408): keep both tests. feat(rocm): add RDNA3 and RDNA4 runtime foundation FlashML-org/FreeToken#132's ROCm gate test and our constexpr AST guard are independent.
utils/hf.py (at fork fix(server): resolve Hugging Face repo ids to a local checkpoint dir #23): imports are re plus Any, FrozenSet, Mapping.
server/openai_api.py and tests/server/test_stream_cancellation.py (at the review-fixes merge): took the original merge's files. Upstream did not touch them between the two bases; this was checked blob by blob.
models/qwen4_exp/ple_disk.py (at fork 5080deploy: run the PLE graph fill on its own thread, submitted before the launch (fixes #65) #67, the filler-thread PLE fill):
- Kept our forward_host_ctx.
- Took fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588's eager double buffer as it auto-merged.
- Dropped fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588's _graph_consumed graph fence. Upstream fills after the launch returns. Here the graph fill runs on the filler thread before its own launch can pass its WAIT, so an event recorded after the launch makes fill k wait on graph k, which waits on fill k.
- The previous graph's read of _graph_pinned is already ordered before the next fill by the readback event the job waits on. 5080deploy: generic fixes from 1080deploy (PLE per-fill readback events, host-tier probes, GDN workspace reserve, server fixes, --embed-weights host off) #77 makes that event per-fill.
- The regression test is test_ple_disk_no_graph_fence.py (5080deploy: generic fixes from 1080deploy (PLE per-fill readback events, host-tier probes, GDN workspace reserve, server fixes, --embed-weights host off) #77). It passes on this resolution and fails with TimeoutError: a fill waited on stream position 2 when the fence is restored.
tests/engine/test_kv_quant_config.py: the rotary fake gains table_positions, which feat(rotary): size the rope table for YaRN and add --hf-overrides FlashML-org/FreeToken#573's max_seq_len reads.

Deploy note. FlashML-org#548 renames the _ple_store extension to _row_store. Run setup.py build_ext --inplace before starting, or the default disk PLE backend fails to import.

Effect on this box (RTX 5080, Qwen3.8-Flash-Next-NVFP4, production config). These figures cover the whole stack (this PR plus #76 and #77):

fix(runtime): correct overlap token limits and cache safety FlashML-org/FreeToken#546 fixes a live bug. Under overlap scheduling, max_tokens=N returned N-1 tokens (2→1, 8→7, 64→63). It now returns exactly N.
feat(rotary): size the rope table for YaRN and add --hf-overrides FlashML-org/FreeToken#573 frees ~80 MiB of VRAM: idle 14642→14562 MiB, 255K-with-side-calls peak 15716→15636 MiB. Its rope-table sizing is the likely cause.
perf(qsa): reduce block top-k candidates in levels past 16 splits FlashML-org/FreeToken#571 is inert at a 262K window. fix(ple): fence disk staging against in-flight lookups FlashML-org/FreeToken#588's eager double buffer fixes a race in eager fills. Our config did not show it, but upstream's GPU test reproduces it on the old code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

…org#546) * fix(scheduler): count delivered tokens under overlap * fix(moe): initialize slots used by inactive hybrid routes * fix(kvcache): report usable window capacity floors * refactor(moe): inline the zeroed slot cache allocation * test: drop the new regression tests --------- Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>

* feat: add simple ROCm GPU support for RDNA3 (gfx1100-1103) - Add hip_compat.h shim mapping CUDA runtime API to HIP equivalents - Update pinned_tensor.cpp to compile under both nvcc and hipcc - Add ROCm detection in arch.py (is_rocm, get_rocm_gfx_arch, is_gfx11xx_family) - Guard NVIDIA arch checks to return None on ROCm - Skip nvcc version check in _toolchain.py when on ROCm - Add ROCm build path in setup.py (ROCM_HOME, amdhip64, --offload-arch) - Add _hip_cflags() in kernel/utils.py for JIT compilation on ROCm - Add is_rocm() and driver_hip_version() in backend.py - Add rocm-smi fallback in __main__.py for clangd generation - Add TODO(ROCm) for NCCL->RCCL, flashinfer/sgl_kernel ROCm builds, Triton autotune RDNA3 tuning, PDL equivalent, hiprtc JIT cache - Add AMD ROCm classifier in pyproject.toml * fix(rocm): fall back for single-bank fast index copy * test(rocm): cover fast index copy fallback * test(rocm): use explicit fast copy module import * fix(rocm): complete RDNA3 runtime path * fix(rocm): preserve CPU MoE implementation * build(rocm): support RDNA4 discovery and modular SDKs * fix(rocm): complete modular SDK runtime support * test(rocm): harden foundation platform contracts * fix(rocm): pin jit target and split install guide * perf(kernel): restore ROCm non-temporal index copy --------- Co-authored-by: bouclem <bouclement2012@gmail.com> Co-authored-by: Yuu <206304251+nekomario28@users.noreply.github.com> Co-authored-by: LZ-QWQ <lz1830837889@163.com>

…ashML-org#571)

…nt.d (FlashML-org#576)

…ashML-org#573) * feat(rotary): size the rope table for YaRN and add --hf-overrides * fix(rotary): guard mrope's table width and copy --hf-overrides values - MRotaryEmbedding asserts its cos/sin table is rotary_dim wide. A proportional rope's table spans the head, and the mrope kernel indexes the rotary_dim // 2 section table with the table's width. - _with_overrides deep-copies every value it stores: transformers fills rope defaults into the rope_parameters it is handed, which was the caller's dict. - The rope-table tests the engine test covers fold into it, the two shared-table tests are one, and the --hf-overrides test moves to tests/utils/test_hf.py, where it loads the override against the same values written into config.json, for a model type transformers knows and one it does not. - docs/cli.md lists --hf-overrides. * fix(server): give --hf-overrides a fresh dict per parse argparse hands its default object to every parse that omits the flag, so default={} put one shared dict in every ServerArgs. The flag defaults to None and becomes a new dict where the arguments are assembled, as --mm-processor-kwargs does. * docs(rotary): an indexer's heads differ in width from the attention's, not wider

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>

# Conflicts: # docs/models.md # python/freetoken/attention/triton.py # python/freetoken/kernel/triton/attention.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # docs/models.md # python/freetoken/attention/triton.py # python/freetoken/kernel/triton/attention.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # docs/models.md # python/freetoken/attention/triton.py # python/freetoken/kernel/triton/attention.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # docs/models.md # python/freetoken/attention/triton.py # python/freetoken/kernel/triton/attention.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # tests/kernels/test_e4m3_compat.py

Assisted-by: Claude Opus 5 (cherry picked from commit 150c95d) (cherry picked from commit 4db8b85) (cherry picked from commit 495770e)

…ol arguments

…ill conv

…n kernel

…d-request # Conflicts: # python/freetoken/engine/config.py # Conflicts: # python/freetoken/engine/config.py # Conflicts: # python/freetoken/engine/config.py

… is measurable # Conflicts: # python/freetoken/engine/engine.py # Conflicts: # python/freetoken/engine/engine.py # Conflicts: # python/freetoken/engine/engine.py

…acked on FlashML-org#385 TP) # Conflicts: # python/freetoken/models/qwen4_exp/attention.py # python/freetoken/models/qwen4_exp/config.py # python/freetoken/models/qwen4_exp/weight.py # Conflicts: # python/freetoken/models/qwen4_exp/attention.py # python/freetoken/models/qwen4_exp/config.py # python/freetoken/models/qwen4_exp/weight.py # Conflicts: # python/freetoken/models/qwen4_exp/attention.py # python/freetoken/models/qwen4_exp/config.py # python/freetoken/models/qwen4_exp/weight.py

… read Assisted-by: Claude Opus 5 (cherry picked from commit 82a21ae) (cherry picked from commit 0369ed9) (cherry picked from commit b5c7812)

… cpu build test Assisted-by: Claude Opus 5 (cherry picked from commit 27ad92e) (cherry picked from commit e152f66) (cherry picked from commit 333f7f9)

…s the quantized-kv backend reads Assisted-by: Claude Opus 5 (cherry picked from commit 600819f) (cherry picked from commit dd2ebdf) (cherry picked from commit 12754d4)

… bound, not an oracle

…lit differs from the fp8 env flags

…when its slot floor does not fit

…wo-stage to fit sm_120 shared memory

…ing ones in a runtime rebuild

…e Hugging Face repo ids to a local checkpoint dir # Conflicts: # python/freetoken/utils/hf.py

…verage # Conflicts: # python/freetoken/tokenizer/tokenize.py # Conflicts: # python/freetoken/engine/sample.py # python/freetoken/tokenizer/tokenize.py # Conflicts: # python/freetoken/engine/sample.py # python/freetoken/tokenizer/tokenize.py # Conflicts: # python/freetoken/engine/sample.py # python/freetoken/tokenizer/tokenize.py

…onfig Fork #19 asks the model family to check the stored fp8 split against the env flags; a synthetic FTW written straight from tensors (upstream's test_ftw_weights) carries no config.json, so there is no family to ask and the read proceeds as before. Assisted-by: Claude Fable 5.1 (cherry picked from commit cab9be3) (cherry picked from commit e59e544)

…oint across prefill chunks

…in zmq pull queues

…ream cancellation # Conflicts: # python/freetoken/server/api_server.py # python/freetoken/server/openai_api.py # Conflicts: # python/freetoken/server/api_server.py # python/freetoken/server/openai_api.py # Conflicts: # python/freetoken/server/api_server.py # python/freetoken/server/openai_api.py # Conflicts: # python/freetoken/server/api_server.py # python/freetoken/server/openai_api.py

…isconnect Guards the FlashML-org#222 resolution against FlashML-org#393's fan-out: the disconnect watcher takes the whole uid list, not the first sample's uid. Assisted-by: Claude Fable 5.1 (cherry picked from commit 93ff96349235ea3ca1504733a2abbd6e6bcccf94) (cherry picked from commit 69a5efa) (cherry picked from commit 40e9941)

…can see a client disconnect Starlette's @app.middleware("http") (BaseHTTPMiddleware) wraps the handler's receive channel and never yields the client's http.disconnect while a non-streaming request is still computing, so request.is_disconnected() stays False and the disconnect watcher from FlashML-org#222 could not stop an abandoned request. Streaming requests only worked because the socket write failed. The recorder is the one such middleware; as pure ASGI it records the same row at response start. Assisted-by: Claude Fable 5.1 (cherry picked from commit 8adde91) (cherry picked from commit 202d7fe)

…ention models (--kv-host-pages) # Conflicts: # python/freetoken/attention/qsa_sparse.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # python/freetoken/attention/qsa_sparse.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py # Conflicts: # python/freetoken/attention/qsa_sparse.py # python/freetoken/kvcache/__init__.py # python/freetoken/kvcache/qsa_pool.py

…ybrid GDN models (snapshots + KV pages)

…cales they cannot mirror Upstream FlashML-org#499/FlashML-org#525 mirror KV codes plus the fp8 pool's per-token _scale_buffer; the nvfp4 pool (FlashML-org#408) keeps per-block scales in _block_scale_buffer, which neither the offloader nor the prefix tier copies, so --kv-host-pages or FT_PREFIX_HOST would rehydrate codes without scales. Both constructors now raise a clear error naming the knob; the GDN-snapshot-only tier (FT_GDN_HOST_TIER) never reads the KV pool and stays usable. Assisted-by: Claude Fable 5.1 (cherry picked from commit 4bb35fc)

…dation without hidden sync

… endpoint

…ics under --enable-metrics # Conflicts: # python/freetoken/scheduler/scheduler.py # python/freetoken/server/generation.py # python/freetoken/server/openai_api.py # Conflicts: # python/freetoken/scheduler/scheduler.py # python/freetoken/server/generation.py # python/freetoken/server/openai_api.py

…etrics timing reads Upstream FlashML-org#504 times the prefill span in Scheduler._prefill_start; the stubs from FlashML-org#222 and FlashML-org#505's tests build the scheduler without it. Assisted-by: Claude Opus 5.5

…hat and legacy completions # Conflicts: # python/freetoken/core.py # python/freetoken/engine/engine.py # python/freetoken/engine/sample.py # python/freetoken/message/frontend.py # python/freetoken/message/tokenizer.py # python/freetoken/scheduler/scheduler.py # python/freetoken/server/generation.py # python/freetoken/server/openai_api.py # python/freetoken/tokenizer/detokenize.py # python/freetoken/tokenizer/server.py # Conflicts: # python/freetoken/core.py # python/freetoken/engine/engine.py # python/freetoken/engine/sample.py # python/freetoken/message/frontend.py # python/freetoken/message/tokenizer.py # python/freetoken/scheduler/scheduler.py # python/freetoken/server/generation.py # python/freetoken/server/openai_api.py # python/freetoken/tokenizer/detokenize.py # python/freetoken/tokenizer/server.py

…lag the sampler plan reads Fork #47's sampler plan asks each request whether it needs logits processing; the upstream FlashML-org#224 test fakes predate it. Assisted-by: Claude Opus 5.5

Assisted-by: Claude Opus 5.5 # Conflicts: # python/freetoken/server/openai_api.py # tests/server/test_stream_cancellation.py

… submitted before the launch # Conflicts: # python/freetoken/models/qwen4_exp/ple_disk.py

…ETOKEN_MOE_SMALL_PREFILL_TOKENS, off by default)

… and hyper-connection projections

A/B on the 5080 (offload, 1650 slots, 32K and 2K contexts, 4 reps each): no decode gain (~38.3 vs ~38.8 tok/s), idle VRAM not reduced, and the quality probe lost one arithmetic item (13/15 vs 14/15). At batch size 1 the bf16 GEMVs already run near memory bandwidth; the FP8 path pads a single row into a _scaled_mm tile and adds a quantize launch per call. The branch perf/fp8-shared-expert-hc stays for reference. Assisted-by: Claude Opus 5.5

…que ids under Triton's element limit

…expert beside the miss copy (FREETOKEN_MOE_COPY_OVERLAP), opt-in slim copy

…writer slot maps (FREETOKEN_MOE_PREFETCH), plus decode verify and slot audit debug modes

…ML-org#573's max_seq_len reads Assisted-by: Claude Opus 5.5

Copilot

Copilot review overview

🟡 Changes recommended

Automatic dtype selection ignores HF overrides, and stale ROCm runtime links can prevent JIT builds.

Review effort: Balanced
Findings: 2

Open (2)

Stale ROCm compatibility symlinks break cold JIT builds · New
HF dtype override ignored when dtype is set to auto · New

What changed in this PR

Rebuilds the stacked 5080 deployment branch on upstream d3512b4, integrating engine fixes, long-context support, and experimental ROCm foundations.

Changes:

Corrects overlap token limits and adds shared disk-row storage with eager PLE double buffering.
Adds HF config overrides, shared YaRN rotary tables, and multi-level QSA top-k reduction.
Introduces ROCm compatibility and updates installation metadata and documentation.

File	Description
tests/utils/test_rocm_arch.py	Tests ROCm target and linker resolution.
tests/utils/test_hf.py	Tests config override semantics.
tests/models/test_qwen3_vl.py	Updates config-loader fake signature.
tests/models/qwen4_exp/test_qsa_kernels.py	Exercises wider top-k reductions.
tests/models/qwen4_exp/test_ple_disk.py	Tests eager staging safety; renames extension imports.
tests/models/qwen4_exp/test_ple_disk_fill_order.py	Updates double-buffer test fakes.
tests/models/qwen4_exp/test_config.py	Tests the extended-context YaRN recipe.
tests/kvcache/test_cache_unit_bytes.py	Adjusts usable window-floor expectation.
tests/kernels/test_row_store.py	Tests generic disk-row storage.
tests/kernels/test_rotary.py	Tests table extension and sharing.
tests/kernels/test_rocm_launch_kwargs.py	Tests ROCm launch arguments.
tests/kernels/test_pinned_tensor.py	Handles HIP pointer-validation differences.
tests/kernels/test_mrope.py	Tests incompatible table-width rejection.
tests/kernels/test_e4m3_compat.py	Adds ROCm compatibility-gate coverage.
tests/engine/test_kv_quant_config.py	Adds table coverage to rotary fake.
tests/engine/test_cache_budget.py	Tests YaRN context-limit checks.
tests/engine/test_attention_backend_matrix.py	Uses the updated rotary config.
setup.py	Adds ROCm builds and renames row-store extension.
python/freetoken/utils/hf.py	Applies checkpoint config overrides.
python/freetoken/utils/arch.py	Resolves ROCm architecture targets.
python/freetoken/utils/__init__.py	Exports ROCm helpers.
python/freetoken/server/args.py	Adds `--hf-overrides`.
python/freetoken/scheduler/scheduler.py	Uses delivered length for token limits.
python/freetoken/moe/offload_cache.py	Zero-initializes expert cache banks.
python/freetoken/models/qwen4_exp/ple_disk.py	Shares row-store helpers; double-buffers eager staging.
python/freetoken/models/config.py	Exposes rotary table position coverage.
python/freetoken/layers/rotary.py	Shares tables and extends YaRN coverage.
python/freetoken/kvcache/cache_status.py	Reports usable window-cache floors.
python/freetoken/kernel/utils.py	Adds ROCm compilation and linking paths.
python/freetoken/kernel/triton/qsa/topk.py	Adds hierarchical candidate reductions.
python/freetoken/kernel/triton/norm.py	Omits CUDA-only launch attributes on ROCm.
python/freetoken/kernel/triton/e4m3_compat.py	Disables NVIDIA-native FP8 selection on ROCm.
python/freetoken/kernel/triton/activation.py	Adds HIP math and launch compatibility.
python/freetoken/kernel/row_store.py	Exposes row storage and graph-sync helpers.
python/freetoken/kernel/pynccl.py	Adds a ROCm migration TODO comment.
python/freetoken/kernel/csrc/row_store/row_store_ext.cpp	Extracts generic storage beneath PLE hashing.
python/freetoken/kernel/csrc/pinned_tensor.cpp	Uses the HIP compatibility header.
python/freetoken/kernel/csrc/jit/fast_index_copy.cuh	Adds ROCm vector-copy support.
python/freetoken/kernel/csrc/include/freetoken/utils.cuh	Disables CUDA launch attributes on ROCm.
python/freetoken/kernel/csrc/include/freetoken/hip_compat.h	Maps runtime APIs to HIP equivalents.
python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp	Uses the HIP compatibility header.
python/freetoken/kernel/backend.py	Adds ROCm detection and driver probing.
python/freetoken/kernel/_toolchain.py	Bypasses NVIDIA toolchain checks on ROCm.
python/freetoken/kernel/__main__.py	Generates HIP-aware editor flags.
python/freetoken/engine/engine.py	Validates context limits against table coverage.
python/freetoken/engine/config.py	Propagates overrides and extended context limits.
python/freetoken/attention/qsa_sparse.py	Reuses the attention rotary table.
pyproject.toml	Declares ROCm support and widens Triton compatibility.
install.sh	Persists `FREETOKEN_HOME` for Desktop integration.
docs/install.md	Links the experimental AMD guide.
docs/install_amd.md	Documents ROCm source installation.
docs/cli.md	Documents HF overrides.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+            link_dir = pathlib.Path.home() / ".cache" / "freetoken" / "rocm-lib"
+            link_dir.mkdir(parents=True, exist_ok=True)
+            compat_link = link_dir / "libamdhip64.so"
+            if not compat_link.exists() and not compat_link.is_symlink():
+                try:
+                    compat_link.symlink_to(versioned[-1])
+                except FileExistsError:
+                    # Multiple tensor-parallel ranks may prepare the same cache.
+                    pass


        image_max_tokens=image_max_tokens,
        processor_kwargs=kwargs.pop("mm_processor_kwargs") or {},
    )
+    kwargs["hf_overrides"] = kwargs["hf_overrides"] or {}


jason-fxz and others added 30 commits September 23, 2026 22:22

refactor(ple): extract a shared disk row store (FlashML-org#548)

f5b9700

perf(qsa): reduce block top-k candidates in levels past 16 splits (Fl…

5031e3d

…ashML-org#571)

feat(install): record the install root as FREETOKEN_HOME in environme…

eba7120

…nt.d (FlashML-org#576)

fix(ple): fence disk staging against in-flight lookups (FlashML-org#588)

d3512b4

Co-authored-by: andy-yang-1 <andy_yang@berkeley.edu>

test(kvcache): give the qsa fp8 pool fakes the mrope flag main now reads

08c223e

Assisted-by: Claude Opus 5 (cherry picked from commit 150c95d) (cherry picked from commit 4db8b85) (cherry picked from commit 495770e)

merge FlashML-org#435: resolve JSON Schema refs and union types in to…

3b3b742

…ol arguments

merge FlashML-org#391: accept the OpenAI developer role

078d14a

merge FlashML-org#475: respect explicit enable_thinking=false with tools

a38a09b

merge FlashML-org#339: drop the D2H sync from the varlen GDN/KDA pref…

dce1579

…ill conv

merge FlashML-org#338: fuse the PLE n-gram row-id hash into one Trito…

e270985

…n kernel

merge FlashML-org#169: load prefill triton kernels at startup, not mi…

0d28e10

…d-request # Conflicts: # python/freetoken/engine/config.py # Conflicts: # python/freetoken/engine/config.py # Conflicts: # python/freetoken/engine/config.py

merge FlashML-org#231: --moe-collect-stats, so expert-cache behaviour…

af1be3d

… is measurable # Conflicts: # python/freetoken/engine/engine.py # Conflicts: # python/freetoken/engine/engine.py # Conflicts: # python/freetoken/engine/engine.py

fix(models): restore the os import and truthy set the fp8 env helpers…

8ff7f4a

… read Assisted-by: Claude Opus 5 (cherry picked from commit 82a21ae) (cherry picked from commit 0369ed9) (cherry picked from commit b5c7812)

test(qwen4_exp): restore the rope device and clear get_rope after the…

a1398c8

… cpu build test Assisted-by: Claude Opus 5 (cherry picked from commit 27ad92e) (cherry picked from commit e152f66) (cherry picked from commit 333f7f9)

test(kernels): give the block_ends backend fake the kv scale accessor…

5dee633

…s the quantized-kv backend reads Assisted-by: Claude Opus 5 (cherry picked from commit 600819f) (cherry picked from commit dd2ebdf) (cherry picked from commit 12754d4)

merge fork #18: fix(moe): report the routing top-k figure as a static…

b616e41

… bound, not an oracle

merge fork #19: fix(qwen4_exp): refuse an ftw checkpoint whose fp8 sp…

340c81b

…lit differs from the fp8 env flags

merge fork #20: fix(engine): let moe-cache-auto drop prefill overlap …

e1cdc87

…when its slot floor does not fit

merge fork #21: fix(triton): keep the float glm dsa sparse launches t…

704bc90

…wo-stage to fit sm_120 shared memory

merge fork #22: fix(engine): resize shrinking cache pools before grow…

e9e63eb

…ing ones in a runtime rebuild

merge fork #23 (upstream FlashML-org#17 rebased): fix(server): resolv…

bdbae96

…e Hugging Face repo ids to a local checkpoint dir # Conflicts: # python/freetoken/utils/hf.py

merge FlashML-org#505: fix(scheduler): preserve pending hybrid checkp…

f79bd0d

…oint across prefill chunks

merge FlashML-org#466: fix(utils): tolerate coalesced msgpack frames …

1777351

…in zmq pull queues

gdevenyi added 22 commits October 1, 2026 23:16

merge FlashML-org#525: feat(kvcache): host-resident prefix tier for h…

3b96582

…ybrid GDN models (snapshots + KV pages)

merge FlashML-org#500: perf(moe): single-launch prefill buffer invali…

fb6b756

…dation without hidden sync

merge FlashML-org#108: feat(server): add OpenAI /v1/models/{model_id}…

61a799b

… endpoint

test(engine): give the logprobs sampler fakes the logits-processing f…

fb6b9a5

…lag the sampler plan reads Fork #47's sampler plan asks each request whether it needs logits processing; the upstream FlashML-org#224 test fakes predate it. Assisted-by: Claude Opus 5.5

merge fork review fixes: 11 bugs found re-reviewing the 5080deploy stack

b7c05d7

Assisted-by: Claude Opus 5.5 # Conflicts: # python/freetoken/server/openai_api.py # tests/server/test_stream_cancellation.py

merge fork fix (issue #65): run the PLE graph fill on its own thread,…

1d195ae

… submitted before the launch # Conflicts: # python/freetoken/models/qwen4_exp/ple_disk.py

merge fork perf: load only the routed experts for short prefills (FRE…

60961c3

…ETOKEN_MOE_SMALL_PREFILL_TOKENS, off by default)

merge fork perf: opt-in post-load FP8 for the qwen4_exp shared expert…

1d20164

… and hyper-connection projections

merge fork fix: small-prefill on-demand path hands the LRU kernel uni…

31d7bc9

…que ids under Triton's element limit

merge fork fix: small-prefill ensures its experts in decode-width chunks

a426033

merge fork perf: scored expert eviction (--moe-cache-policy), shared …

f2bd437

…expert beside the miss copy (FREETOKEN_MOE_COPY_OVERLAP), opt-in slim copy

merge fork perf: expert prefetch into low-priority slots with single-…

9afbb4c

…writer slot maps (FREETOKEN_MOE_PREFETCH), plus decode verify and slot audit debug modes

test(engine): give the kv quant config fake the rope table size Flash…

43e0cad

…ML-org#573's max_seq_len reads Assisted-by: Claude Opus 5.5

Copilot AI balanced review requested due to automatic review settings October 2, 2026 05:51

Copilot started reviewing on behalf of gdevenyi October 2, 2026 05:52

View session

gdevenyi mentioned this pull request Oct 2, 2026

5080deploy 9/9: merge upstream #389 (+#385), load-time FP8 dense projections #17

Open

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

5080deploy: rebuild on upstream main d3512b4 (upstream #546 max_tokens fix, #548 row store, #588 eager PLE buffers without its graph fence) - #75

5080deploy: rebuild on upstream main d3512b4 (upstream #546 max_tokens fix, #548 row store, #588 eager PLE buffers without its graph fence)#75
gdevenyi wants to merge 60 commits into
5080/base-8ba20f6from
5080/33-rebuild-d3512b4

gdevenyi commented Oct 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

gdevenyi commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gdevenyi commented Oct 2, 2026 •

edited

Loading