Spare 1-D norm weights from a blanket --type - #1861
Open
danielhanchen wants to merge 1 commit into
Open
Conversation
tensor_should_be_converted has no rule for 1-D weights, so a per-channel norm
scale survives a blanket --type only by accident: when its length does not
divide the quant block size (FLUX q_norm/k_norm are [128] and 128 % 256 != 0),
or when one of the FLUX-era name rules happens to match it.
MiniMax-H3's per-block norms are [5376], and 5376 % 256 == 0, so a blanket
--type q4_K quantizes 105 of them. The result loads and renders a plausible
video, so a "does it run" check passes it, but scored against a bf16 render of
the same prompt and seed it is destroyed.
A 1-D weight is a per-channel gain, never a matmul weight. Every channel of a
block would share one scale and one min, and a gain vector has no reason to be
locally smooth, so quantizing it buys almost nothing and costs a lot: on this
model holding all 211 1-D tensors adds 0.77 MiB to a 10.60 GiB file, 0.007%.
Measured on minimax_h3_fl2va_pruned, converted from the same bf16 checkpoint
with a blanket --type q4_K and no --tensor-type-rules, rendered at 640x384,
25 frames, 4 steps, cfg 1.0, seed 1234, --rng cpu, and scored against a bf16
render of that same prompt and seed:
1-D tensor types PSNR SSIM LPIPS
before F32 5, F16 51, BF16 106, Q4_K 105 9.87 0.074 0.981
after F32 5, F16 51, BF16 211 22.22 0.841 0.292
SSIM 0.074 means the output is essentially uncorrelated with the reference.
The 1-D rule alone is sufficient. A build made with
--tensor-type-rules 'norm[0-9]*\.weight$=bf16,condition_proj\.weight$=bf16'
produces the same 1-D layout and scores 21.52 / 0.838 / 0.299, so nothing here
depends on also sparing condition_proj.weight, which is 2-D and a separate
quality choice. Peak VRAM is unchanged at 16.83 GiB.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24a4aff97b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1522
to
+1525
| // crushed to 4 bits by a blanket --type. The result still loads and still renders a | ||
| // plausible image, so a "does it run" check passes it, while measured against a bf16 | ||
| // render of the same prompt and seed it is destroyed: PSNR 9.87 / SSIM 0.074 / | ||
| // LPIPS 0.981, against 22.22 / 0.841 / 0.292 with this rule in place. |
There was a problem hiding this comment.
Remove the model-specific history and PSNR/SSIM/LPIPS results from this comment: these non-reproducible measurements document this particular change rather than an enduring constraint and will become misleading as models or quantizers evolve. Keep only the invariant that quantizing one-dimensional gain vectors damages quality; the detailed evidence belongs in the commit description, documentation, or a regression test.
AGENTS.md reference: AGENTS.md:L127-L134
Useful? React with 👍 / 👎.
Comment on lines
+1515
to
+1516
| // Pass, do not convert. A 1-D weight is a per-channel scale (LayerNorm/RMSNorm gain), | ||
| // never a matmul weight, so quantizing it buys almost nothing and costs a lot: every |
There was a problem hiding this comment.
Correct the claim that every one-dimensional weight is a normalization gain: this repository also creates one-dimensional learned class embeddings (src/model/te/clip.hpp), pad tokens (src/model/diffusion/z_image.hpp), and modulation offsets (src/model/diffusion/krea2.hpp). The new condition intentionally exempts all of these from quantization, so documenting a narrower invariant misstates the behavior and may cause a maintainer to incorrectly replace the shape check with a norm-name check.
AGENTS.md reference: AGENTS.md:L129-L132
Useful? React with 👍 / 👎.
| if (type != GGML_TYPE_COUNT) { | ||
| if (ggml_is_quantized(type) && tensor_storage.ne[0] % ggml_blck_size(type) != 0) { | ||
| // Pass, do not convert | ||
| } else if (ggml_is_quantized(type) && tensor_storage.n_dims <= 1) { |
There was a problem hiding this comment.
Do not use TensorStorage::n_dims here as proof that a GGUF tensor is logically one-dimensional. src/model_io/gguf_io.cpp populates it with ggml_n_dims(dummy), which drops a trailing singleton dimension, so a real 2-D matmul such as Krea2's Linear(text_layers, 1) projector is read back with n_dims == 1 and silently excluded from a requested blanket quantization. This makes GGUF-to-GGUF conversion or runtime --type handling depend on the input container rather than the tensor's logical shape.
AGENTS.md reference: AGENTS.md:L178-L180
Useful? React with 👍 / 👎.
danielhanchen
added a commit
to unslothai/stable-diffusion.cpp
that referenced
this pull request
Aug 8, 2026
The prebuilt pipeline builds leejet's source at an aged release tag, not this fork's master, so the three MiniMax-H3 fixes on master reach nobody: every Studio user installs a binary that still aborts on the default cfg-scale, still aborts on --vae-on-cpu, and still quantizes H3's 1-D norms into an output uncorrelated with its own bf16 reference. Building from master instead would throw away the reason the pipeline is shaped this way, which is that what we publish should be traceable to a specific upstream release. So keep the upstream tag as the base and carry the delta explicitly: - patches/ holds one file per fix, each with its upstream pull request in the header. All three are open on leejet: leejet#1861, leejet#1862, leejet#1863. - resolve applies them to the checked-out tag, after running git apply --check over the whole set so a stale patch stops the run before the tree is half modified. That failure is the signal to delete the patch (upstream merged it) or refresh it (upstream moved the code). - a non-empty set moves the published tag to <upstream tag>-u<id>, where id is the sha256 prefix of the concatenated patches. The tag then says whether a box is stock, and a changed patch set republishes rather than matching an existing release and skipping. - the manifest and the release notes both record the applied list. An empty patches/ leaves the tag and every asset name exactly as they are today. Verified by running the resolve step against master-813-bfbef5b with gh stubbed: all three patches apply, the tag becomes master-813-bfbef5b-u<id>, and the stamped source tarball contains the fixes.
danielhanchen
added a commit
to oobabooga/unsloth
that referenced
this pull request
Aug 8, 2026
The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3 does not work: it aborts on the default --cfg-scale, aborts again on --vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by pinning cfg to 1.0 and the second by dropping the flag; the third had no workaround on the consumer side at all. All three are fixed in unslothai/stable-diffusion.cpp and open upstream as leejet/stable-diffusion.cpp#1861, unslothai#1862 and unslothai#1863. The mirror's prebuilt pipeline now applies them on top of the aged upstream tag it already builds, and marks such a build with a -u<id> suffix naming the patch set, so master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three patches and nothing else. The patches are deleted once upstream releases them, at which point this pin goes back to a plain tag. Verified on the published Linux x86_64 asset, not on a local build: both new error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser without --mode vid_gen now exits 1 with the instruction instead of core dumping on a ggml assert. test_video_backend's fake engine returned the old tag as its version string, which read like a second pin; it only needs a non-None value, so it now says so.
danielhanchen
added a commit
to unslothai/unsloth
that referenced
this pull request
Aug 9, 2026
* Add MiniMax H3 video generation
* Improve MiniMax H3 memory routing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address MiniMax H3 review feedback
* Keep H3's VAE off the CPU path, so low_vram stops aborting
low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it
unconditionally. On H3 that kills the process:
ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed
deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with
--audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an
fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video
one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the
file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c),
ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and
audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and
lets F32 through. It cannot be fixed by shipping a different checkpoint.
low_vram is the one mode a small-card user reaches for, so this drops the flag
rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other
family changes, and the H3 native path passes False. --offload-to-cpu and
--clip-on-cpu still apply, which is where the saving actually is: the denoiser
dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Point the H3 GGUF pick at the unsloth mirror
Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires
every curated video gguf_repo to live under unsloth/ so a one-click pick
cannot 404 when a community repack is renamed or taken down. H3 was the
one family still on a community repo.
This was meant to be part of the merge commit but was left in the
worktree, so CI on that commit still saw the old value.
unsloth/MiniMax-H3-GGUF is still private and has to be made public
before this merges, or the pick will 401. No CI check reads it.
* Pin H3's native cfg-scale under test
H3 is distilled and CFG-free: its empty unconditional prompt encodes to
zero tokens, and the transposed tensor that produces trips
GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134.
Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp
defaults cfg-scale to 7.0, so this is a crash a plausible refactor
reintroduces by forwarding guidance the way every other family does.
The native path already hardcodes 1.0 and the family sets
supports_cfg = False, but nothing held either in place. supports_cfg
only gates the diffusers path; the native path builds its own params.
Checked the test fails when cfg_scale is changed to forward guidance,
so it is not passing vacuously.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin diffusers by source archive instead of git, so macOS installs work
The macos-15-intel leg fails deterministically, not flakily:
process didn't exit successfully: `/usr/bin/git init` (exit status: 1)
--- stderr
xcode-select: note: No developer tools were found, requesting install.
That runner has no Xcode, so /usr/bin/git is the developer-tools shim and
exits 1 for everything. uv needs a working git to resolve a git+https
dependency, so the diffusers pin this branch added cannot install there
at all. Main is unaffected because it depends on plain diffusers.
GitHub serves the same commit as a source archive, which uv installs with
no git involved. Verified by stubbing git to fail exactly the way the
macOS shim does: the git+ form reproduces the CI error, the archive form
installs diffusers 0.40.0.dev0 from the same SHA with
MiniMaxH3Transformer3DModel present and exported.
Also drops a full clone of diffusers from every install.
* Allow the diffusers source build on the clean-machine legs
With the archive pin the macOS leg gets past `git init` and installs, but
then trips the nobuild guard:
built from source: diffusers -- these must resolve to wheels on a clean machine
There is no wheel to resolve to. MiniMax-H3 support is not in any
diffusers release, so this branch has to pin a commit, and neither a git
URL nor a source archive can produce a wheel from an index.
Added to the same allowlist that already carries triton-kernels for the
same reason. Checked against the bar the comment there sets: diffusers
builds with plain setuptools, declares no ext_modules, and its tree has
zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517
build is a pure-Python copy step and needs no toolchain. The separate
compiler-invocation check in both scripts is untouched and still fires if
one is ever needed.
Verified the allowlist logic still rejects a non-allowlisted source build
(a log building both diffusers and numpy reports only numpy).
Remove this entry once a diffusers release carries H3 and the requirement
goes back to a version specifier. Noted in both scripts.
* Baseline the hf-hub retry loop reopened by the 1.x upgrade
`pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined
CRITICAL:
C2 polling/beaconing loop detected
Package: huggingface-hub
File: huggingface_hub/utils/_http.py
L461: while True:
This is on us, not upstream drift. Main pins huggingface-hub==0.36.2;
this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit
requires it, so the resolved version moves 0.36.2 -> 1.27.0.
The baseline already carries this exact file and check at L462 and L298,
from earlier versions. It did not carry over because the key hashes the
matched code, not the line number, and the surrounding code changed
across the major version. That is the guard behaving correctly: changed
code in a baselined file reopens for review rather than staying
suppressed.
Reviewed it rather than just re-suppressing. L461 is the retry loop in
`http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from
the caller, there is no hardcoded endpoint and nothing is exfiltrated.
Same benign construct as the entries it replaces.
Entry generated with the scanner's own _evidence_hash rather than
hand-written, and inserted beside its siblings so the diff stays 8 lines.
Verified: with the baseline the scan is exit 0 with 4 suppressed, without
it exit 1, so the guard still bites.
* Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable
`mac macos-15 / trace / file` fails on this branch and passes on main. uv
reports:
No solution found when resolving dependencies:
Because you require huggingface-hub>=0.34.0,<1.0 and
huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable.
This branch moved base.txt, no-torch-runtime.txt, studio.txt and
constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned
commit requires it, but left the flat <1.0 cap in
overrides-darwin-arm64.txt. That file is macOS-arm only, which is why
only the mac legs see it and Linux and Windows stayed green.
The failure then presents as something else entirely: uv gives up, the
installer falls back to pip, and the clean-machine trace fails on
"installer invoked toolchain: rustc" rather than on the resolution.
The cap's own comment explains it exists so the resolver can never pair
hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is
still true below python 3.10 and still capped there. At 3.10 and above
this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone,
and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway.
Checked by collecting every hub specifier that applies per Python version
across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to
1.23.0/1.27.0. Before this change py3.13 resolved to nothing.
* Make H3's native download use the repo the family advertises
The curated-mirror test main added only inspects VideoFamily.gguf_repo.
H3's native path does not read that field: video_minimax_h3.py has its
own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL
encoder. So pointing the family at the unsloth mirror in the previous
commit satisfied that test while the actual one-click download still came
from a community repack, which is the exact failure the test exists to
prevent.
Pointed the constant at the same mirror and added
test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so
the two cannot drift apart again. Verified it fails when the constant is
put back to leejet, so it is not passing vacuously.
The mirror now carries the Qwen3-VL encoder quants alongside the
denoisers, byte-identical in size to the community ones, so this repo
alone satisfies both of h3_native_hub_files' hub entries. The encoder is
part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already
mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping
a quantization of it beside the denoisers is the same act.
Also checked the encoder-tier routing survives the dynamic rung names:
-UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one,
asserted in the new test.
Updated the download-plan test, which hardcoded the old repo id.
NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really
does gate this: the native path downloads from it. It has to be public
before this merges.
* Pin H3's companion-checkpoint guard under test
validate_h3_transformer_filename had no test. That mattered less when the
denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in
the same repo, so the picker lists both and a user can name either. Loading
a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead
of at the boundary.
The accept cases include the dynamic rung names on purpose. The guard is a
prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it
was written; it happens to pass, and now that is asserted rather than
assumed.
Checked the test fails when the prefix check is dropped, so it is not
vacuous.
* Record why H3 drops --vae-on-cpu, now that the abort is fixed
The comment justified the drop entirely by an sd.cpp abort. That abort is fixed
in the Unsloth fork, which would have made this look like a stale workaround to
revert once the fix reaches the pinned prebuilt.
Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with
--offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak
VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu
the peak is set by the streamed denoiser, so the flag saves nothing and costs
4.8x. It stays off on its own merits.
* Pin the sd.cpp prebuilt that actually renders MiniMax-H3
The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3
does not work: it aborts on the default --cfg-scale, aborts again on
--vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with
its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by
pinning cfg to 1.0 and the second by dropping the flag; the third had no
workaround on the consumer side at all.
All three are fixed in unslothai/stable-diffusion.cpp and open upstream as
leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt
pipeline now applies them on top of the aged upstream tag it already builds, and
marks such a build with a -u<id> suffix naming the patch set, so
master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three
patches and nothing else. The patches are deleted once upstream releases them,
at which point this pin goes back to a plain tag.
Verified on the published Linux x86_64 asset, not on a local build: both new
error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser
without --mode vid_gen now exits 1 with the instruction instead of core dumping
on a ggml assert.
test_video_backend's fake engine returned the old tag as its version string,
which read like a second pin; it only needs a non-None value, so it now says so.
* Close two gaps the H3 mirror switch opened
Both are consequences of the two preceding commits, found in review.
The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and
_resolve_with_fallback still asked leejet for it. That request is a guaranteed
404 by construction, since the -u<id> suffix marks a build only the mirror
makes, so it was a wasted round trip on every install. Worse, when the mirror
genuinely cannot serve a host the fallback lands on leejet's latest, which has
none of the H3 fixes. For every other model that is the right trade, better a
stock native engine than none. For H3 it is not visible: it aborts on the
default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken
video rather than failing. A user who saw only the generic 'falling back to
leejet' line had nothing connecting that to the output. It now says so.
Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo
id only. The mirror the family and catalog now advertise carries the Qwen3-VL
encoder quants beside the denoisers, so a 12 GB text encoder was being
aggregated as if it were a selectable transformer quant. Both bundle repos are
now recognised, case-insensitively, and the cache-dir match follows.
Tests are mutation-verified rather than assumed: restoring the upstream 404
attempt fails the ordering test, removing the H3 warning fails the fallback
test, and dropping the mirror from the bundle set fails three. The ordering test
deliberately makes the mirror serve nothing, because with the mirror serving the
first attempt succeeds and the upstream attempts are never reached, which made
an earlier version of it pass under its own mutation.
A third test pins the native loader's H3_GGUF_REPO to the bundle set, since
those live in different files and a future repo move that updated only the
loader would silently reintroduce the same aggregation bug.
* Exclude MiniMax-H3's small-M projections from int8
_INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5
but none for minimax-h3, and H3 needs one for the same reason they do.
H3's adaLN projection is named adaln_proj, which no token in the generic list
matches: 'norm' is the closest and does not appear in the name. On the dense
checkpoint that projection is Linear(2688 -> 96768), so it clears
min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8
matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be
greater than 16, but got 1' at the first denoise. The offline prequant builder
bakes it in happily, which is exactly the drift the exclude list exists to
prevent for Flux and Qwen.
The pruned-modulation form hides this rather than fixing it: there adaln_proj is
Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what
makes the DENSE path correct and is a no-op on the pruned one.
context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5
excludes its text stream. Measured at M = 10 text tokens against the video
stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path,
so leaving them bf16 costs nothing measurable.
This is what made an earlier measurement conclude int8 does not work on H3. It
does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at
identical memory, paired over 12 renders.
* Say which H3 component could not be downloaded, and why
H3 pulls four files from two repos, and the Hub returns the same
'Repository Not Found ... make sure you are authenticated' for a repo that does
not exist, one that is private, and one your token does not cover. A user
reading that has no way to tell which of the four failed, and the wording points
away from the real cause whenever the repo exists but is not public.
That is the state the GGUF mirror is in today: it is unpublished, so picking H3
fails with a message suggesting the user fix their token, which will not help.
This replaces it with the repo, the component, and the actual remedy, and says
the other components are unaffected so the failure is not read as total.
Gated repos get different wording, since accepting a licence is a different
action from waiting for a repo to be published.
Anything that is not a recognised access error is passed back unchanged rather
than reworded, so a timeout or a full disk still reads as itself. The helper
returns the exception instead of raising, so the caller keeps raise-from and the
original traceback survives.
Mutation-verified three ways: rewording every error (timeouts included), fixing
the component name to 'denoiser', and giving gated repos the private wording
each fail the test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop MiniMax-H3 holding two copies of its video VAE
Two thirds of an H3 render's peak is not activations. Measured at
640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB,
and the gap is almost all weights: the video VAE alone is 10.42 GB
because diffusers pins it to float32, and a further 4.91 GB is
autocast's own float16 copy of those same weights. A memory snapshot
puts 92.9% of the transient in 437 blocks allocated from nn.Linear,
largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection
in float16.
MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16),
and autocast caches every weight it casts for the lifetime of the
region, so the float32 original and its float16 twin are both resident
through the whole decode. Storing those weights as float16 up front
makes the cast a no-op and removes both. This is not an approximation:
x.to(float16).to(float16) is x.to(float16), and the four regression
tests check that on a real matmul rather than on the reasoning. The
audio VAE decode is not under autocast, so it keeps float32.
t2va starts from noise and never encodes, so vae.encoder and
vae.quant_conv go too. That part is gated on the workflow name rather
than dropped unconditionally, because an image-conditioned workflow
needs them.
Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast,
28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed
is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every
arm hashes identically to its control on latents, audio_latents, frames
and audio.
The estimator's base still reads 68.5 GB. That figure was measured on
the bfloat16 modular components, not the int8 arm above, so it stays put
until it is re-measured in the same configuration rather than adjusted
by arithmetic.
* Pad MiniMax-H3's small-M int8 linears instead of leaving them dense
torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips
it (safe_int_mm falls back to a widened matmul), but inductor lowers the
same quantized linear straight to _int_mm, so any quantized Linear invoked
at a small activation row count crashes under torch.compile. Until now the
fix was to leave those linears dense bf16, which on H3 meant excluding
context_embedder and both token_refiner blocks: 13 linears, 798M
parameters, 0.80 GB of weights the int8 checkpoint was not allowed to
touch.
Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up
to 32, runs the GEMM and slices the result back, so the module compiles
with no change to the quantization config and the caller's rows come back
bitwise unchanged. Verified bitwise on all 65 (module, M) cases across
H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized
weights; compiling those same modules unpadded raises the _int_mm assert
at M = 10, 13 and 14.
Two properties carry that exactness and both are asserted rather than
assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax
0, so the activation quantizer divides by zero. And the activation scale
must be per row, so each kept row's scale comes from that row alone; a
quantized Linear whose granularity cannot be proven per row raises instead
of being quietly skipped, because a half-padded transformer compiles on
the wrapped modules and crashes on the rest.
Everything below pad_to normalises to pad_to rather than only what is
below the floor, so one inductor graph covers every prompt length in the
range. H3's seven eval prompts run at M = 10..19, which straddles the
floor, and padding only to 17 would leave three shapes behind.
The wrapper reparents the Linear, so it runs after quantize_ on the
runtime path and after load_state_dict on the prequant one. The offline
builder drives quantize_ directly and saves the state dict, so it never
sees a wrapper; PadToMinM is also state-dict transparent as a second line
of defence, saving and loading under its own prefix so a wrapped
transformer still writes context_embedder.weight.
Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5
have the same small-M shape but published int8 prequant checkpoints whose
metadata bakes the current exclusion set, and _validate_checkpoint
compares that set against exclude_tokens_for_scheme, so they move only
together with a rebuild. adaln_proj stays excluded for a different reason:
on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1,
while on the pruned form it is Linear(8 -> 96768) and falls under
min_features anyway (verified: the filter rejects all 51 for min_features).
Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the
two arms alternated within each cell so drift on a shared box cancels:
checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%)
transformer 21.051 -> 20.253 GB (-0.798)
load peak 21.137 -> 20.336 GB (-0.801)
render peak 37.766 -> 36.966 GB (-0.800, -2.1%)
step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled
The time is torchao's un-fused eager activation quantization on 13 modules
that run ONCE per render, so it is a fixed cost rather than a per-step
one, and it does not scale with steps or resolution. Compiling those
modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable
difference. Quality is unchanged as far as n = 14 can resolve: against the
same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the
excluded one, which rules out a degradation larger than 0.022.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Classify the MiniMax-H3 GGUF bundle as video in the cached inventory
The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata
keys at all (kv_count 0), so general.architecture is absent where the
LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left
the downloaded repo with no task, which drops it from the Video picker's
On Device list and hands it to chat as a plain GGUF. Key the two bundle
repo ids before the arch is consulted.
* Release the VIDEO GPU claim when H3 native falls back to the CPU build
On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the
resolved device target is not CPU. _run_load_h3_native then asks for an
accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux
CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the
CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next
chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM.
Release the claim once the CPU fallback is committed, through release_if so the
token check is atomic against a newer load that already took ownership. Mirrors
the CPU-only native release /images/load already does.
* Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token
- VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime
re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard
consults it, so deleting a companion under a loaded model is refused.
- The H3 modular workflow no longer falls back to a null progress context: the denoise
loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the
same per-step progress and cancellation the other callback-less pipeline gets.
- load_components() gets the Settings token, so gated/private component loads are not
issued anonymously.
* Reject stable-diffusion.cpp builds that predate MiniMax-H3 support
ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only
probes that it runs, so an install upgraded from an older Studio kept serving
its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(),
which that binary passes, so the load reported ready and the failure surfaced
on the first generation, after the whole bundle had downloaded.
Gate the H3 path alone on the capability instead of a version string: upstream
added --ref-video and the other H3-only options in the same commit that added
MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the
release prebuilts report 'version unknown, commit unknown' because they are
built without a .git directory, so --help is the only usable signal. Image
generation keeps accepting any user-supplied build.
A stale copy under the installer-owned root is removed so the pinned prebuilt
reinstalls; a build the user supplied is left in place and the load fails
naming it, the same ownership split _usable_or_discard_managed makes. A --help
that cannot be read means 'cannot tell', never 'no H3'.
* Distinguish a reused CPU sd-cli from an accelerator build on an H3 load
On a Linux CUDA host the first H3 load installs the CPU prebuilt through the
fallback and correctly commits native_device = cpu, because the pinned tag
publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls
ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and
returns it without looking at what it was built for, so the fallback was
skipped, native_device stayed cuda, and Studio applied GPU offload policy and
retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A
later chat or image acquire then evicted an unrelated GPU model.
Fall back on what the binary offers rather than on whether one was returned:
sd-cli --list-devices prints one name/description line per available ggml
backend device, so a CPU-only build answers with CPU alone. The second load now
reaches the same cpu conclusion as the first, which is what lets the existing
release_if drop the stale claim. Unreadable output, or an older build that
rejects the flag, keeps the GPU: neither says the accelerator is missing.
* Guard the H3 companion repos while a native video load is downloading
_run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3
component repo, but the in-flight state only carried repo_id and base_repo. The
cached-model delete guard reads loading_repo_ids(), so it allowed deleting
Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another
mirror or a local file, while those files were still downloading, which fails
the load.
Carry the companions on the loading state, the way the image backend's
_SdLoading already does, and publish them from loading_repo_ids(). This is the
in-flight twin of loaded_repo_ids() and covers the same repos.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep curated Recommended rows searchable and give them their metadata
Two things went wrong with the curated video models the Recommended list
paints from the catalog rather than from a live Hub listing.
Search dropped them. The Recommended search matched the query against
`recommendedIds`, which filters out every id already on disk because a
downloaded model gets its own On Device row. The unfiltered Recommended
list does not filter that way: it renders the curated seeds and badges the
downloaded ones. So a curated model was visible in the list and unfindable
by typing its name the moment it was downloaded, and only a live listing
row could bring it back, which a repo the listing does not return never
gets. `searchableRecommendedIds` unions the seed ids with the listing ids,
seeds first, deduped case insensitively (the HF cache lowercases repo ids),
so both lists agree on what exists.
Rows rendered bare. Everything past the id came from the listing alone, so
a curated row the listing never returns showed no parameter chip and no
capability glyph while its neighbours showed both. The catalog now carries
the two facts nothing else can supply: `totalParams` per artifact and
`capabilities` per group, read through `curatedTotalParamsFor` and
`curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins
wherever there is one, because real tags and a Hub-reported total outrank
anything hand written here.
The parameter counts are measured, not guessed. The MiniMax-H3 figure is
the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that
repo publishes; the LTX-2.3 figure is what the Hub reports for its repo,
carried so the row looks the same offline or rate limited.
One more inconsistency fell out of the same place: `searchRowFits` hides
anything it cannot size (`requireKnown`), and it could not size a curated
repo with no listing row and no "<n>B" token in its id, so turning on
"Fits on device" hid from search a row the unfiltered list still showed.
It now falls back to the curated total the same way it already fell back
to the curated size.
Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts:
15 assertions over the search pool, the two catalog lookups, the fit check,
and the four picker call sites that read them.
* Load MiniMax-H3 from a hosted pre-quantized denoiser
Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily
carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in
video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full
66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with
DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that
shape rather than inventing a second one.
Four parts.
VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder,
resolved by video_family_prequant_repo() / video_family_prequant_schemes(),
mirroring the image resolver. The registries stay separate, as the module header
requires, so the base-id normaliser is local rather than imported.
The shared resolver learns an optional subfolder. The hosted video checkpoints
nest theirs one level down instead of keeping it at the repo root, and the prefix
has to reach BOTH candidate names or the primary 404 is followed by a second one
and the load silently falls back to the dense download. Always a literal forward
slash: these are Hub repo paths, and a Windows join would miss the cache. The
cache and download plumbing already handled a nested name, so only the filename
builder changed, and every existing call site is byte-identical.
The modular workflow builds its denoiser through its own component loader, so
there is no dense module to quantise in place. A hosted checkpoint is therefore
the only way to run that transformer quantized, and pre-seeding it with
update_components() before load_components() is also what stops the dense
download: load_components(names=None) skips a component whose attribute is
already set. Passing names= instead would have forfeited the workflow's own block
pruning, which is what already avoids the 61.7 GB Ref2VA transformer.
Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine
factorization of the time-embedding curve plus a shared table, and roughly 40% of
the released model's parameters go. Against the base repo's dense config the
model is four keys short, one over and fifty-one shapes wrong, so the strict load
fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py
reshapes the model between from_config and load_state_dict: table lookup with
interpolation instead of the timestep MLP, no SiLU (the table already holds the
activation's own output projected onto the basis), and the modulation cast to the
block stack's dtype, without which the first quantized matmul dies on mismatched
dtypes. Bound per instance, so a dense load in the same process is untouched.
Finally the refusal becomes honest. A single-file pick on a modular family used
to reach the loader only after ~98.7 GB had downloaded AND after the resident
pipeline had been evicted to make room for it, because download-plan returned 200
and validation passed. Both refusals now run in validate_load_request, ahead of
the diffusers availability probe so they still fire where diffusers cannot be
imported, and each names what to pick instead. download-plan forwards
transformer_quant to validation and to the plan, without which the quant-keyed
refusal never fires on the route that stages the download and the plan stages
shards the load never opens.
* Point both MiniMax-H3 schemes at one hosted pre-quantized repo
The two hosted pre-quantized denoisers were split across two repos with the
checkpoint nested one level down, so reaching them needed a mechanism the image
side has never had: a VideoFamily.prequant_subfolder field, a
prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through
both resolve_prequant_source() and usable_prequant_source().
Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt.
That is the layout every image-side prequant repo already uses, and it is exactly
what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match
an existing convention and the code should shrink, not grow: -22 lines in
diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new
concept to carry.
Landing on the primary name also fixes a memory-planning under-credit.
cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that
a cached legacy artifact cannot pin a stale name once a repo ships the real one.
While these checkpoints were published as transformer_<scheme>.pt the primary
never existed: every hit came through fallback_filename, and planning therefore
read an already-cached checkpoint as "this would have to download" and handed the
pick to GGUF. The primary is now the published name, so the probe hits it.
fallback_filename stays. It still covers repos that have not been renamed, and
dropping it is a separate decision from this one.
Tests: the subfolder-prefix cases are replaced by the naming they now guard --
both schemes resolving to one repo, the primary resolving to a root-level
<Model>-<SCHEME>.pt with no directory component on any platform, the cache probe
being asked for that primary name, and the repo's own scheme suffix being
stripped and replaced rather than carried through. Five mutations run, each
caught by the named test and reverted: two repos again (M1), the primary nested
under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed
on the fallback (M4), the fallback name dropped (M5).
* Add MiniMax-H3 image and reference video conditioning
* Improve H3 finalization progress
* Report real sd.cpp progress on the Video page instead of a frozen 0 of 30
A native (GGUF) video generation reported phase "denoise", step 0/30 for its
entire run and then flipped straight to "completed". Two separate things were
wrong, and the progress endpoint could not move until both were fixed.
The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling
bar contains neither word. It prints
|=========> | 7/30 - 21.50s/it
so nothing ever matched. Anchor the pattern on the bar and on the trailing
speed unit instead. A bare "n/m" is deliberately not enough: an unrelated
ratio in some other log line must not drive the progress bar.
The reader also delivered every redraw one step late. sd-cli redraws in
place, and its carriage return LEADS the record while the newline only
arrives on the final step, so a reader keyed on CR/LF cannot produce step 1
until step 2 has been flushed. Treat the erase-to-end-of-line that closes
each redraw as a terminator too, and read the pipe with buffer.read1 so a
record that carries no newline is not stuck behind a blocking readline.
Escapes are stripped before a record reaches on_log or the error tail.
Streams without a raw .buffer keep the old line iteration.
The same bar shape is printed by three different things, so the parser now
tells them apart rather than reporting whichever came last. Weight load
prints it with a byte rate, and tiled VAE decode prints an identical s/it bar
counting TILES: without a guard a run finished sampling at 30/30 and then
jumped backwards to "step 1/16". Load and decode are real work with no
sampling step, so they report their own phase and a null step rather than a
fake 0 of 30. ETA is measured from the first step, not from job start, so the
one-off weight load is not charged to every remaining step.
Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the
step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are
reported as themselves, and the ETA tracks.
* Read sd-cli's in-place progress redraws so the Video bar moves during sampling
The native H3 progress bar had two independent causes and fixing either alone
changed nothing observable. The bar pattern is now correct, but the reader still
was not.
sd-cli redraws its sampling bar in place: one printf per step shaped
"\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a
phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only,
so every redraw sat in the buffer until the next one's carriage return arrived
and the last one until sampling was already over. The Video page saw nothing.
Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line,
reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte
character straddling two reads survives, and strip the CSI escapes before the
record reaches on_log or the error tail. Streams without a raw .buffer (test
doubles, non-pipes) fall back to line iteration.
The new backend test drives the real byte stream through both halves, one flush
per read, and asserts each step is visible on the read that carried it rather
than one redraw later.
* Give MiniMax-H3 first and last frame conditioning in the video backend
MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same
checkpoint run with no keyframes. Studio only ever ran it text-only, so the
Video generate request had nowhere to attach a reference frame.
The load used to prune the block graph to t2va. That argument prunes STATICALLY,
so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot
serve a text-only one at all: it raises packing an empty conditioning list.
The load now keeps the whole auto graph, which selects per request, and bounds
load_components to the keyframe workflow's component set instead, so the 61.7 GB
Ref2VA partition is no more loaded than before. Measured against the released
checkpoint: a text-only request through this pipeline is bit-identical, video
and audio, to the same request through a t2va-pruned one.
A keyframe is a geometry anchor, not just conditioning, so the canvas comes from
its aspect ratio through the released arithmetic (768 short edge, area capped at
768x1344, both axes rounded to 32) rather than from whatever resolution preset
was selected. An arbitrary size produces a garbled clip rather than an error.
sd.cpp already implements the same conditioning, so that side is the existing
--init-img / --end-img flags with the frames staged as PNGs.
Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX
do not grow a control that does nothing.
* Cover the MiniMax-H3 keyframe path with tests
Registry (which families declare it, and the canvas a keyframe resolves to),
the load wiring (whole block graph, component set still bounded, VAE encoder
kept), request handling (decode, refusal, canvas override, what reaches the
pipeline call) and the sd-cli argv.
* Add the reference-frame controls to the Video page
First and last frame pickers, shown only for a family whose status declares
keyframe conditioning, so Wan and LTX are unchanged. The Images page's source
picker moves to a shared component rather than the Video page growing a second
upload path; both send the same data URL to the same backend decoder.
While a frame is attached the Resolution preset is disabled and says why: the
frame's aspect ratio resolves the clip's size, the way the model itself does.
The gallery recipe records which ends were pinned.
* Check the keyframe canvas against the pipeline's own resolver
The canvas rule is a checkpoint contract, so pin it to the released
implementation rather than only to hand-written expectations. Skipped where
diffusers does not ship MiniMax-H3, which is most runners.
* Revert the standalone H3 keyframe implementation
oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and
the canvas rule, and it reached the same load construction independently. Two
implementations of the same feature on one branch is worse than either, so this
takes mine back out and leaves the branch ready for that work to land whole.
The one finding worth keeping from it is already reflected there: passing
workflow= to ModularPipeline.from_pretrained prunes the block graph statically,
so the pipeline must be built unpruned and only load_components bounded.
* Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run
Loading H3 with a hosted pre-quantized denoiser worked, but every generation died
on its first denoise step:
Attempted to set the storage of a tensor on device "cuda:0" to a storage on
different device "cpu". This is no longer allowed; the devices must match.
ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and
moves each one onto the accelerator inside its own pre_forward, that is from
within the block that is already executing. The text encoder and the VAEs survive
that; a torchao-quantized denoiser does not, because the device change reaches
return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator
tensor. Moving the same module at load time, outside any executing block, works.
So place it once at load and take it out of the offload rotation: drop its hook,
unlist it from the other components' eviction candidates, and move it. Everything
else is unchanged, and the encoder and VAEs still offload around it. Keeping it
resident is what asking for a quantized denoiser buys in the first place: the
hosted checkpoint is about 20 GB against 66.3 GB dense.
Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8
denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s.
The clip's first frame matches the supplied image and the motion is coherent.
* Apply the pinned Diffusers revision on a fresh install, not just an update
MiniMax-H3 needs a Diffusers revision newer than any published release, and
Studio refuses to load it otherwise. The pin was in
studio/backend/requirements/base.txt, and a clean install.sh run still ended up
on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so.
base.txt is never installed by install.sh. install.sh installs unsloth itself,
whose own metadata pulls a diffusers release in transitively, and then runs
install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is
a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch
path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly
the path a new user takes. Reproduced on a clean install into a throwaway prefix
before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel
present.
The revision now lives in its own diffusers-pin.txt, installed by a step that
sits outside every skip_base / NO_TORCH branch and after every other
requirements file, so nothing left in the run can re-resolve diffusers back to a
release behind it. No forced reinstall is needed: a direct URL requirement is not
satisfied by a resident registry install, so the step is a no-op once the
environment is already on the pin.
tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one
requirements file may name diffusers, the pin must be a full commit sha rather
than a moving ref, the install step must sit at function top level rather than
under a conditional, and it must come after every other requirements install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let the Video and Images pickers see a local diffusers pipeline
A model already on disk never reached the pickers' On Device list unless it
happened to keep a weight file beside a root config.json. Every image and video
model downloaded as a pipeline keeps its weights in component subdirs under a
root model_index.json instead, so the hub inventory scan behind /api/hub/local
rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio
publisher walk descended into the pipeline and offered vae, transformer,
text_encoder and audio_vae as four separate models, none of them loadable.
Teach the hub scanner the same pipeline-root test routes/models.py already
applies, in the three places that only accepted a root config plus loose
weights, and keep a pipeline row through the custom-folder format filter: the
layout has no loose weight to classify, so the row is "unknown" by construction
rather than by fault.
* Cover the local diffusers pipeline scan with tests
* Pin what the pipeline exemption must not let through
Three gaps in the cover added with the scan change, each found by mutating the fix and
watching the suite stay green.
The custom-folder format filter now waves a row through on its shape, and nothing said
what it still has to reject: replacing the whole predicate with True passed. A folder
holding a config.json and no weights, which an aborted download leaves behind, reports
the same "unknown" format and no loader can start it, so it pins the boundary.
The predicate is applied to every row the filter did not already accept, and a row's
path can be a GGUF file rather than a directory. A missing path, a file, and a
directory whose model_index.json is itself a directory must answer False rather than
raise, because an exception there fails the scan and empties the picker.
The publisher walk was only covered one level up. Adding the model folder itself as a
scan folder is the obvious thing to do and used to publish vae, transformer and
text_encoder as three models.
* Drop the unused H3_TASK_KEYFRAMES import from the video backend
video.py only branches on H3_TASK_REFERENCES; the keyframe constant is read
from video_minimax_h3 directly by the tests that need it. The hoisted-import
safety net in Source lint flags the unused name as a blocker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stage the hosted pre-quantized H3 denoiser in the video download plan
The plan already drops the dense transformer shards whenever a hosted
pre-quantized checkpoint covers them, but nothing put that checkpoint back:
an int8 or fp8 H3 stage skipped 66.3 GB of base shards and added none of the
20.25 GB artifact the load actually opens. The byte total under-reported the
stage by the size of the checkpoint, the disk preflight cleared a volume that
could not hold it, and an offline stage completed without the one file the
load needs.
_denoiser_prequant_hub_files mirrors the pre-cast encoder helper: it resolves
the family's hosted checkpoint, confirms the file really exists on the Hub,
and prefers the repo-root name over the legacy scheme name in the same order
the load tries them. An unreachable repo is logged and yields no files, so a
gated or renamed artifact keeps the dense shards instead of sinking the plan.
The progress-bar estimate is deliberately left alone: it counts cached bytes
for the checkpoint and base repos only, so adding a third repo there would
leave the bar permanently short of 100 percent.
* List both H3 denoiser partitions in the picker, not only FL2VA
The bundle-repo filter accepted only minimax_h3_fl2va*, so every published
minimax_h3_ref2va* quant was hidden from the variant listing. The loader
disagrees: validate_h3_transformer_filename accepts either partition, on the
grounds that which one is picked IS the task, and h3_transformer_task routes
Ref2VA to the reference-video workflow this PR adds. The community bundle repo
publishes three Ref2VA quants today, so the reference path was unreachable from
the remote catalog.
Accept both prefixes from one shared tuple and keep excluding the Qwen3-VL
encoder and VAE companions, which are never picks for either partition. The
filter test asserted the old behaviour and is updated with it.
* Bound H3 reference-audio decoding to the trained window
The reference-video decoder already selects, resizes and refuses incrementally
because the encoded size says nothing about the decoded size. The audio decoder
did not: it appended every resampled block to a list and then allocated a second
full buffer in np.concatenate, with no duration or sample cap.
The route accepts 32 MiB of encoded audio, which is over half an hour of
compressed stereo. That lands as roughly 1.9 GB of float32 and doubles again in
the concatenate, and up to three references are accepted per request, so an
ordinary long music or podcast file picked by mistake could exhaust the host
before the background job even started.
H3's reference window is 15 seconds, so anything past it is unusable rather than
merely large. Refuse it while decoding, with the same shape of message the video
guard uses, instead of decoding it first.
* Refuse a quantized H3 reference load instead of seeding the keyframe denoiser
The hosted pre-quantized checkpoints are FL2VA (keyframe) denoisers. Ref2VA
shares their module shapes and the same base model, so resolve_prequant_source
handed one back for a reference load, it passed every metadata check, and
seeding it made load_components skip the real Ref2VA transformer. The request
then generated from the wrong partition rather than failing, which is the worst
of the three outcomes.
The route accepts h3_task, so this was reachable from the public API even though
the picker does not expose the choice yet. validate_load_request now refuses the
pairing with a message naming the workable alternatives, in the same place it
already refuses a scheme with no hosted checkpoint, and the modular loader drops
to the released components if a direct call reaches it. Nothing changes for
keyframe loads, which are what the checkpoints are.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip modular-workflow families in the dense text-encoder plan assertion
The plan-unchanged sweep from main walks every family and asserts the dense
budget plan_diffusion_memory received. MiniMax-H3 is the first modular-workflow
family to reach that list, and load_pipeline dispatches to the workflow's own
loader before the planner runs: each component is built by its own
from_pretrained, so there is no single dense pipeline to budget and no plan call
to assert on. Skip it the way the sweep already skips wan2.2-t2v-a14b.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the paired-axes canvas rule to keyframe requests
A half-specified width/height is long-standing, documented API behaviour: the backend
resolves the missing axis from the family's default preset, in both
validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule
as an unconditional request validator rejected those calls with a 422 before family
validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients.
The rule still holds where it means something: with a keyframe present the canvas is
matched to the source aspect whenever either axis is missing, so the axis the caller
sent would be silently discarded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the CUDA sd-cli pin and translate a mirror-only tag upstream
The merge with main brings in the accelerator-aware installer, whose fallback
translates a mirror-only -u<id> pin back to the upstream release it was built
from instead of skipping the upstream attempt. That is strictly better: skipping
kept the round trip cheap but dropped the pin entirely on every host the mirror
does not build, leaving them on upstream latest.
test_a_mirror_only_pin_is_never_requested_upstream asserted the old shape, that
the fallback settled for upstream latest. It now pins the new one: never the
literal -u<id> string, the translated release instead, and no latest attempt at
all because the translated pin succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
danielhanchen
added a commit
to unslothai/unsloth
that referenced
this pull request
Aug 10, 2026
* Add MiniMax H3 video generation
* Improve MiniMax H3 memory routing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address MiniMax H3 review feedback
* Keep H3's VAE off the CPU path, so low_vram stops aborting
low_vram maps to the `model` policy, and offload_flags emits --vae-on-cpu for it
unconditionally. On H3 that kills the process:
ggml/src/ggml-cpu/ops.cpp:6321: GGML_ASSERT(src0->type == GGML_TYPE_F16) failed
deterministically, SIGABRT, exit 134. Bisected on the flags: --vae-on-cpu with
--audio-vae aborts, the same command without --audio-vae renders in 95.87s, and an
fp16-converted audio VAE aborts too. So the trigger is the audio VAE, not the video
one, and the F32 type is imposed inside stable-diffusion.cpp rather than by the
file: ggml_conv_1d hardcodes an F16 im2col destination (ggml/src/ggml.c),
ggml_compute_forward_im2col_f16 then asserts the KERNEL is F16, and
audio_conv_weight_type (src/model/vae/ltx_audio_vae.hpp) maps only BF16 to F16 and
lets F32 through. It cannot be fixed by shipping a different checkpoint.
low_vram is the one mode a small-card user reaches for, so this drops the flag
rather than the mode. offload_flags takes vae_on_cpu, defaulting True so no other
family changes, and the H3 native path passes False. --offload-to-cpu and
--clip-on-cpu still apply, which is where the saving actually is: the denoiser
dominates, and with --offload-to-cpu the whole model peaks at 13.14 GiB.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Point the H3 GGUF pick at the unsloth mirror
Main added test_curated_gguf_repos_are_unsloth_mirrors, which requires
every curated video gguf_repo to live under unsloth/ so a one-click pick
cannot 404 when a community repack is renamed or taken down. H3 was the
one family still on a community repo.
This was meant to be part of the merge commit but was left in the
worktree, so CI on that commit still saw the old value.
unsloth/MiniMax-H3-GGUF is still private and has to be made public
before this merges, or the pick will 401. No CI check reads it.
* Pin H3's native cfg-scale under test
H3 is distilled and CFG-free: its empty unconditional prompt encodes to
zero tokens, and the transposed tensor that produces trips
GGML_ASSERT(!ggml_is_transposed(a)) in ggml.c. SIGABRT, exit 134.
Measured: cfg 1.0 renders, cfg 1.5 and cfg 4.0 both abort. sd.cpp
defaults cfg-scale to 7.0, so this is a crash a plausible refactor
reintroduces by forwarding guidance the way every other family does.
The native path already hardcodes 1.0 and the family sets
supports_cfg = False, but nothing held either in place. supports_cfg
only gates the diffusers path; the native path builds its own params.
Checked the test fails when cfg_scale is changed to forward guidance,
so it is not passing vacuously.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin diffusers by source archive instead of git, so macOS installs work
The macos-15-intel leg fails deterministically, not flakily:
process didn't exit successfully: `/usr/bin/git init` (exit status: 1)
--- stderr
xcode-select: note: No developer tools were found, requesting install.
That runner has no Xcode, so /usr/bin/git is the developer-tools shim and
exits 1 for everything. uv needs a working git to resolve a git+https
dependency, so the diffusers pin this branch added cannot install there
at all. Main is unaffected because it depends on plain diffusers.
GitHub serves the same commit as a source archive, which uv installs with
no git involved. Verified by stubbing git to fail exactly the way the
macOS shim does: the git+ form reproduces the CI error, the archive form
installs diffusers 0.40.0.dev0 from the same SHA with
MiniMaxH3Transformer3DModel present and exported.
Also drops a full clone of diffusers from every install.
* Allow the diffusers source build on the clean-machine legs
With the archive pin the macOS leg gets past `git init` and installs, but
then trips the nobuild guard:
built from source: diffusers -- these must resolve to wheels on a clean machine
There is no wheel to resolve to. MiniMax-H3 support is not in any
diffusers release, so this branch has to pin a commit, and neither a git
URL nor a source archive can produce a wheel from an index.
Added to the same allowlist that already carries triton-kernels for the
same reason. Checked against the bar the comment there sets: diffusers
builds with plain setuptools, declares no ext_modules, and its tree has
zero .c/.cpp/.pyx/.rs/.cu files and no shipped binaries, so the PEP 517
build is a pure-Python copy step and needs no toolchain. The separate
compiler-invocation check in both scripts is untouched and still fires if
one is ever needed.
Verified the allowlist logic still rejects a non-allowlisted source build
(a log building both diffusers and numpy reports only numpy).
Remove this entry once a diffusers release carries H3 and the requirement
goes back to a version specifier. Noted in both scripts.
* Baseline the hf-hub retry loop reopened by the 1.x upgrade
`pip scan-packages :: hf-stack` fails on this branch with 1 unbaselined
CRITICAL:
C2 polling/beaconing loop detected
Package: huggingface-hub
File: huggingface_hub/utils/_http.py
L461: while True:
This is on us, not upstream drift. Main pins huggingface-hub==0.36.2;
this branch needs >=1.23.0,<2.0 because diffusers at the pinned commit
requires it, so the resolved version moves 0.36.2 -> 1.27.0.
The baseline already carries this exact file and check at L462 and L298,
from earlier versions. It did not carry over because the key hashes the
matched code, not the line number, and the surrounding code changed
across the major version. That is the guard behaving correctly: changed
code in a baselined file reopens for review rather than staying
suppressed.
Reviewed it rather than just re-suppressing. L461 is the retry loop in
`http_backoff`: bounded by `nb_tries > max_retries`, the URL comes from
the caller, there is no hardcoded endpoint and nothing is exfiltrated.
Same benign construct as the entries it replaces.
Entry generated with the scanner's own _evidence_hash rather than
hand-written, and inserted beside its siblings so the diff stays 8 lines.
Verified: with the baseline the scan is exit 0 with 4 suppressed, without
it exit 1, so the guard still bites.
* Lift the macOS-arm huggingface-hub cap that this branch made unsatisfiable
`mac macos-15 / trace / file` fails on this branch and passes on main. uv
reports:
No solution found when resolving dependencies:
Because you require huggingface-hub>=0.34.0,<1.0 and
huggingface-hub>=1.23.0,<2.0, your requirements are unsatisfiable.
This branch moved base.txt, no-torch-runtime.txt, studio.txt and
constraints.txt to hub >=1.23.0,<2.0, because diffusers at the pinned
commit requires it, but left the flat <1.0 cap in
overrides-darwin-arm64.txt. That file is macOS-arm only, which is why
only the mac legs see it and Linux and Windows stayed green.
The failure then presents as something else entirely: uv gives up, the
installer falls back to pip, and the clean-machine trace fails on
"installer invoked toolchain: rustc" rather than on the resolution.
The cap's own comment explains it exists so the resolver can never pair
hub 1.x with a pinned transformers 4.57.6 / hub 0.36.2 stack. That is
still true below python 3.10 and still capped there. At 3.10 and above
this branch is on transformers 5.5.0 and hub 1.x, so the premise is gone,
and mlx-audio's own >=1.0 floor is satisfied by the 1.x window anyway.
Checked by collecting every hub specifier that applies per Python version
across all five files: py3.9 resolves to 0.36.2 as intended, py3.13 to
1.23.0/1.27.0. Before this change py3.13 resolved to nothing.
* Make H3's native download use the repo the family advertises
The curated-mirror test main added only inspects VideoFamily.gguf_repo.
H3's native path does not read that field: video_minimax_h3.py has its
own H3_GGUF_REPO constant, used for both the transformer and the Qwen3-VL
encoder. So pointing the family at the unsloth mirror in the previous
commit satisfied that test while the actual one-click download still came
from a community repack, which is the exact failure the test exists to
prevent.
Pointed the constant at the same mirror and added
test_the_h3_native_repo_matches_the_family_gguf_repo to pin the pair, so
the two cannot drift apart again. Verified it fails when the constant is
put back to leejet, so it is not passing vacuously.
The mirror now carries the Qwen3-VL encoder quants alongside the
denoisers, byte-identical in size to the community ones, so this repo
alone satisfies both of h3_native_hub_files' hub entries. The encoder is
part of MiniMaxAI/MiniMax-H3 itself (FL2VA/text_encoder) which we already
mirror publicly at unsloth/MiniMax-H3 under the same licence, so shipping
a quantization of it beside the denoisers is the same act.
Also checked the encoder-tier routing survives the dynamic rung names:
-UD-Q2_K_XL selects the Q2_K_M encoder and -UD-Q3_K_XL the Q4_K_M one,
asserted in the new test.
Updated the download-plan test, which hardcoded the old repo id.
NOTE: unsloth/MiniMax-H3-GGUF is private. Unlike before, that now really
does gate this: the native path downloads from it. It has to be public
before this merges.
* Pin H3's companion-checkpoint guard under test
validate_h3_transformer_filename had no test. That mattered less when the
denoisers lived alone; the mirror now ships the Qwen3-VL encoder quants in
the same repo, so the picker lists both and a user can name either. Loading
a 12-17 GB encoder as the transformer would fail deep inside sd-cli instead
of at the boundary.
The accept cases include the dynamic rung names on purpose. The guard is a
prefix/suffix check and `-UD-Q2_K_XL` is a shape it had never seen when it
was written; it happens to pass, and now that is asserted rather than
assumed.
Checked the test fails when the prefix check is dropped, so it is not
vacuous.
* Record why H3 drops --vae-on-cpu, now that the abort is fixed
The comment justified the drop entirely by an sd.cpp abort. That abort is fixed
in the Unsloth fork, which would have made this look like a stale workaround to
revert once the fix reaches the pinned prebuilt.
Measured on a build carrying the fix, 640x384, 25 frames, 4 steps, q4_K, with
--offload-to-cpu --clip-on-cpu already applied: adding --vae-on-cpu moved peak
VRAM 12.42 -> 12.42 GiB and wall time 20.9s -> 100.4s. Under --offload-to-cpu
the peak is set by the streamed denoiser, so the flag saves nothing and costs
4.8x. It stays off on its own merits.
* Pin the sd.cpp prebuilt that actually renders MiniMax-H3
The pin was master-812-ea7f0c8, a stock upstream build, and on a stock build H3
does not work: it aborts on the default --cfg-scale, aborts again on
--vae-on-cpu, and its 1-D norms are quantized into an output uncorrelated with
its own bf16 reference (LPIPS 0.981). The Studio side worked around the first by
pinning cfg to 1.0 and the second by dropping the flag; the third had no
workaround on the consumer side at all.
All three are fixed in unslothai/stable-diffusion.cpp and open upstream as
leejet/stable-diffusion.cpp#1861, #1862 and #1863. The mirror's prebuilt
pipeline now applies them on top of the aged upstream tag it already builds, and
marks such a build with a -u<id> suffix naming the patch set, so
master-813-bfbef5b-u0665242 is upstream master-813-bfbef5b plus those three
patches and nothing else. The patches are deleted once upstream releases them,
at which point this pin goes back to a plain tag.
Verified on the published Linux x86_64 asset, not on a local build: both new
error strings are in the shipped sd-cli, and running it on a q4_K H3 denoiser
without --mode vid_gen now exits 1 with the instruction instead of core dumping
on a ggml assert.
test_video_backend's fake engine returned the old tag as its version string,
which read like a second pin; it only needs a non-None value, so it now says so.
* Close two gaps the H3 mirror switch opened
Both are consequences of the two preceding commits, found in review.
The prebuilt pin is now mirror-only (master-813-bfbef5b-u0665242), and
_resolve_with_fallback still asked leejet for it. That request is a guaranteed
404 by construction, since the -u<id> suffix marks a build only the mirror
makes, so it was a wasted round trip on every install. Worse, when the mirror
genuinely cannot serve a host the fallback lands on leejet's latest, which has
none of the H3 fixes. For every other model that is the right trade, better a
stock native engine than none. For H3 it is not visible: it aborts on the
default cfg-scale, aborts on --vae-on-cpu, and a blanket --type renders a broken
video rather than failing. A user who saw only the generic 'falling back to
leejet' line had nothing connecting that to the output. It now says so.
Second, hub/utils/gguf.py filtered H3 companion GGUFs by the old community repo
id only. The mirror the family and catalog now advertise carries the Qwen3-VL
encoder quants beside the denoisers, so a 12 GB text encoder was being
aggregated as if it were a selectable transformer quant. Both bundle repos are
now recognised, case-insensitively, and the cache-dir match follows.
Tests are mutation-verified rather than assumed: restoring the upstream 404
attempt fails the ordering test, removing the H3 warning fails the fallback
test, and dropping the mirror from the bundle set fails three. The ordering test
deliberately makes the mirror serve nothing, because with the mirror serving the
first attempt succeeds and the upstream attempts are never reached, which made
an earlier version of it pass under its own mutation.
A third test pins the native loader's H3_GGUF_REPO to the bundle set, since
those live in different files and a future repo move that updated only the
loader would silently reintroduce the same aggregation bug.
* Exclude MiniMax-H3's small-M projections from int8
_INT8_FAMILY_EXCLUDE_NAME_TOKENS has entries for qwen-image and hunyuanvideo-1.5
but none for minimax-h3, and H3 needs one for the same reason they do.
H3's adaLN projection is named adaln_proj, which no token in the generic list
matches: 'norm' is the closest and does not appear in the name. On the dense
checkpoint that projection is Linear(2688 -> 96768), so it clears
min_features = 512, gets quantized, and then runs at M = 1. Inductor lowers int8
matmul to _int_mm, which requires M > 16, so it raises 'self.size(0) needs to be
greater than 16, but got 1' at the first denoise. The offline prequant builder
bakes it in happily, which is exactly the drift the exclude list exists to
prevent for Flux and Qwen.
The pruned-modulation form hides this rather than fixing it: there adaln_proj is
Linear(8 -> 96768) and falls under min_features anyway. So this exclusion is what
makes the DENSE path correct and is a no-op on the pruned one.
context_embedder and token_refiner are added for the same reason hunyuanvideo-1.5
excludes its text stream. Measured at M = 10 text tokens against the video
stream's thousands, they are 3.47% of GEMM time even in the slow eager int8 path,
so leaving them bf16 costs nothing measurable.
This is what made an earlier measurement conclude int8 does not work on H3. It
does: on the pruned form int8 compiles and is 4.24% +-0.54% faster than fp8 at
identical memory, paired over 12 renders.
* Say which H3 component could not be downloaded, and why
H3 pulls four files from two repos, and the Hub returns the same
'Repository Not Found ... make sure you are authenticated' for a repo that does
not exist, one that is private, and one your token does not cover. A user
reading that has no way to tell which of the four failed, and the wording points
away from the real cause whenever the repo exists but is not public.
That is the state the GGUF mirror is in today: it is unpublished, so picking H3
fails with a message suggesting the user fix their token, which will not help.
This replaces it with the repo, the component, and the actual remedy, and says
the other components are unaffected so the failure is not read as total.
Gated repos get different wording, since accepting a licence is a different
action from waiting for a repo to be published.
Anything that is not a recognised access error is passed back unchanged rather
than reworded, so a timeout or a full disk still reads as itself. The helper
returns the exception instead of raising, so the caller keeps raise-from and the
original traceback survives.
Mutation-verified three ways: rewording every error (timeouts included), fixing
the component name to 'denoiser', and giving gated repos the private wording
each fail the test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop MiniMax-H3 holding two copies of its video VAE
Two thirds of an H3 render's peak is not activations. Measured at
640x384 across 124 frames, a 20.25 GB int8 denoiser peaks at 36.96 GB,
and the gap is almost all weights: the video VAE alone is 10.42 GB
because diffusers pins it to float32, and a further 4.91 GB is
autocast's own float16 copy of those same weights. A memory snapshot
puts 92.9% of the transient in 437 blocks allocated from nn.Linear,
largest 67.1 MB, which is the decoder's [2048, 16384] SwiGLU projection
in float16.
MiniMaxH3VideoDecodeStep wraps vae.decode in torch.autocast(float16),
and autocast caches every weight it casts for the lifetime of the
region, so the float32 original and its float16 twin are both resident
through the whole decode. Storing those weights as float16 up front
makes the cast a no-op and removes both. This is not an approximation:
x.to(float16).to(float16) is x.to(float16), and the four regression
tests check that on a real matmul rather than on the reasoning. The
audio VAE decode is not under autocast, so it keeps float32.
t2va starts from noise and never encodes, so vae.encoder and
vae.quant_conv go too. That part is gated on the workflow name rather
than dropped unconditionally, because an image-conditioned workflow
needs them.
Measured 36.96 -> 28.37 GB peak with the encoder drop and the pre-cast,
28.27 with expandable_segments as well, over 5 prompts x 2 seeds. Speed
is unchanged (-0.05% +-0.86% eager, -0.34% +-2.23% compiled), and every
arm hashes identically to its control on latents, audio_latents, frames
and audio.
The estimator's base still reads 68.5 GB. That figure was measured on
the bfloat16 modular components, not the int8 arm above, so it stays put
until it is re-measured in the same configuration rather than adjusted
by arithmetic.
* Pad MiniMax-H3's small-M int8 linears instead of leaving them dense
torch._int_mm asserts self.size(0) > 16. torchao's eager path never trips
it (safe_int_mm falls back to a widened matmul), but inductor lowers the
same quantized linear straight to _int_mm, so any quantized Linear invoked
at a small activation row count crashes under torch.compile. Until now the
fix was to leave those linears dense bf16, which on H3 meant excluding
context_embedder and both token_refiner blocks: 13 linears, 798M
parameters, 0.80 GB of weights the int8 checkpoint was not allowed to
touch.
Pad instead. diffusion_quant_pad.PadToMinM pads the flattened row count up
to 32, runs the GEMM and slices the result back, so the module compiles
with no change to the quantization config and the caller's rows come back
bitwise unchanged. Verified bitwise on all 65 (module, M) cases across
H3's 13 linears at M = 10, 13, 14, 17, 19, on real torchao-quantized
weights; compiling those same modules unpadded raises the _int_mm assert
at M = 10, 13 and 14.
Two properties carry that exactness and both are asserted rather than
assumed. The pad rows replicate row 0, not zeros: an all-zero row has amax
0, so the activation quantizer divides by zero. And the activation scale
must be per row, so each kept row's scale comes from that row alone; a
quantized Linear whose granularity cannot be proven per row raises instead
of being quietly skipped, because a half-padded transformer compiles on
the wrapped modules and crashes on the rest.
Everything below pad_to normalises to pad_to rather than only what is
below the floor, so one inductor graph covers every prompt length in the
range. H3's seven eval prompts run at M = 10..19, which straddles the
floor, and padding only to 17 would leave three shapes behind.
The wrapper reparents the Linear, so it runs after quantize_ on the
runtime path and after load_state_dict on the prequant one. The offline
builder drives quantize_ directly and saves the state dict, so it never
sees a wrapper; PadToMinM is also state-dict transparent as a second line
of defence, saving and loading under its own prefix so a wrapped
transformer still writes context_embedder.weight.
Scoped to minimax-h3. qwen-image, qwen-image-edit and hunyuanvideo-1.5
have the same small-M shape but published int8 prequant checkpoints whose
metadata bakes the current exclusion set, and _validate_checkpoint
compares that set against exclude_tokens_for_scheme, so they move only
together with a rebuild. adaln_proj stays excluded for a different reason:
on the dense checkpoint it is Linear(2688 -> 96768) and runs at M = 1,
while on the pruned form it is Linear(8 -> 96768) and falls under
min_features anyway (verified: the filter rejects all 51 for min_features).
Measured on B200, 640x384 x 124 frames, 4 steps, 7 prompts x 2 seeds, the
two arms alternated within each cell so drift on a shared box cancels:
checkpoint 21.052 -> 20.254 GB (-0.798, -3.8%)
transformer 21.051 -> 20.253 GB (-0.798)
load peak 21.137 -> 20.336 GB (-0.801)
render peak 37.766 -> 36.966 GB (-0.800, -2.1%)
step time +0.0597 s +-0.0057 eager, +0.0533 s +-0.0086 compiled
The time is torchao's un-fused eager activation quantization on 13 modules
that run ONCE per render, so it is a fixed cost rather than a per-step
one, and it does not scale with steps or resolution. Compiling those
modules alongside the blocks removes it: +0.0034 s +-0.0040, no detectable
difference. Quality is unchanged as far as n = 14 can resolve: against the
same bf16 twin the padded arm sits +0.0070 +-0.0151 LPIPS from the
excluded one, which rules out a degradation larger than 0.022.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Classify the MiniMax-H3 GGUF bundle as video in the cached inventory
The H3 GGUFs are stable-diffusion.cpp conversions and carry no metadata
keys at all (kv_count 0), so general.architecture is absent where the
LTX-2 and Wan video GGUFs declare ltxv/wan. _arch_to_task therefore left
the downloaded repo with no task, which drops it from the Video picker's
On Device list and hands it to chat as a plain GGUF. Key the two bundle
repo ids before the arch is consulted.
* Release the VIDEO GPU claim when H3 native falls back to the CPU build
On a CUDA/ROCm host /video/load acquires the VIDEO arbiter owner because the
resolved device target is not CPU. _run_load_h3_native then asks for an
accelerator-matched sd-cli, and the pinned prebuilt release publishes no Linux
CUDA/ROCm archive, so ensure_sd_cpp_binary returns None and the load commits the
CPU build with native_device = cpu. Nothing dropped the VIDEO claim, so the next
chat or image acquire evicted and unloaded an H3 runtime that holds no VRAM.
Release the claim once the CPU fallback is committed, through release_if so the
token check is atomic against a newer load that already took ownership. Mirrors
the CPU-only native release /images/load already does.
* Video: protect the native H3 companion repos, cancel the modular denoise, forward the hub token
- VideoBackend.loaded_repo_ids() publishes the repos the committed native H3 runtime
re-reads every generation (Qwen encoder + both VAEs), and the delete-cached guard
consults it, so deleting a companion under a loaded model is refused.
- The H3 modular workflow no longer falls back to a null progress context: the denoise
loop drives pipe.scheduler.step once per step, so the existing wrapper gives it the
same per-step progress and cancellation the other callback-less pipeline gets.
- load_components() gets the Settings token, so gated/private component loads are not
issued anonymously.
* Reject stable-diffusion.cpp builds that predate MiniMax-H3 support
ensure_sd_cpp_binary hands back whatever find_sd_cpp_binary locates and only
probes that it runs, so an install upgraded from an older Studio kept serving
its pre-H3 managed sd-cli. The H3 load's only gate is SdCppEngine.version(),
which that binary passes, so the load reported ready and the failure surfaced
on the first generation, after the whole bundle had downloaded.
Gate the H3 path alone on the capability instead of a version string: upstream
added --ref-video and the other H3-only options in the same commit that added
MiniMax-H3 (leejet/stable-diffusion.cpp#1854, master-812-ea7f0c8), and the
release prebuilts report 'version unknown, commit unknown' because they are
built without a .git directory, so --help is the only usable signal. Image
generation keeps accepting any user-supplied build.
A stale copy under the installer-owned root is removed so the pinned prebuilt
reinstalls; a build the user supplied is left in place and the load fails
naming it, the same ownership split _usable_or_discard_managed makes. A --help
that cannot be read means 'cannot tell', never 'no H3'.
* Distinguish a reused CPU sd-cli from an accelerator build on an H3 load
On a Linux CUDA host the first H3 load installs the CPU prebuilt through the
fallback and correctly commits native_device = cpu, because the pinned tag
publishes no Linux CUDA, ROCm or Vulkan asset. Every later load then calls
ensure_sd_cpp_binary(accelerator = cuda), which finds that same CPU binary and
returns it without looking at what it was built for, so the fallback was
skipped, native_device stayed cuda, and Studio applied GPU offload policy and
retained the VIDEO gpu_arbiter claim while sd-cli ran wholly on the CPU. A
later chat or image acquire then evicted an unrelated GPU model.
Fall back on what the binary offers rather than on whether one was returned:
sd-cli --list-devices prints one name/description line per available ggml
backend device, so a CPU-only build answers with CPU alone. The second load now
reaches the same cpu conclusion as the first, which is what lets the existing
release_if drop the stale claim. Unreadable output, or an older build that
rejects the flag, keeps the GPU: neither says the accelerator is missing.
* Guard the H3 companion repos while a native video load is downloading
_run_load_h3_native downloads from repo_id, the H3 GGUF companion and the H3
component repo, but the in-flight state only carried repo_id and base_repo. The
cached-model delete guard reads loading_repo_ids(), so it allowed deleting
Comfy-Org/MiniMax-H3, and the GGUF companion when the load comes from another
mirror or a local file, while those files were still downloading, which fails
the load.
Carry the companions on the loading state, the way the image backend's
_SdLoading already does, and publish them from loading_repo_ids(). This is the
in-flight twin of loaded_repo_ids() and covers the same repos.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: LTX-2 video LoRA training from still images
Adds LTX-2 as the first video family of the flow-matching DiT LoRA trainer. It is
a seventh _FamilySpec plus routing, not a new trainer: the loop, the latent cache,
the phased load and the adapter export are all shared with the image families.
Milestone one trains from a STILL-IMAGE dataset. A 1-frame clip is a valid LTX-2
input (its VAE compresses time by 8), and style and character LoRAs converge on
20 to 50 stills, so discover_image_caption_pairs is reused verbatim and nothing
decodes a video. Clip datasets are a follow-up.
What a video family needed beyond the image path:
- Audio. Every LTX-2 block carries a second audio stream plus two cross-modality
attentions, and forward takes audio_hidden_states as a required argument.
Lightricks handle a video-only run by passing audio=None, which their
transformer short-circuits; diffusers has no such escape hatch, so the
equivalent here is a minimal placeholder audio stream at the step's own sigma,
isolate_modalities=True so it cannot reach the video prediction, no audio in
the loss, and no audio modules in the LoRA targets.
- LoRA targets are fully qualified (attn1.* / attn2.*). The shared bare suffixes
would also match audio_attn1, audio_attn2, audio_to_video_attn and
video_to_audio_attn, which is the mistake Lightricks warn about in their docs.
- Conditioning is two stages: Gemma3-12B, then a connectors module. Only the
connector output is cached (the encoder hidden states are ~50x larger), and
_free_text_encoders now also drops connectors, audio_vae and vocoder.
- LTX-2 conditions on the unscaled timestep, unlike the image families' /1000.
- flow_shift defaults to auto: its scheduler sets use_dynamic_shifting, so
scheduler.sigmas is the unshifted table and training on it would draw a sigma
distribution inference never uses.
Routing: video bases live in a separate registry, so until now every video
checkpoint fell through resolve_trainable_family's unknown-name fallback and was
handed to the SDXL trainer. Video families are now resolved explicitly, and one
without a spec (Wan, HunyuanVideo-1.5) is refused by name.
Also adds the family defaults from the upstream LoRA configs (rank/alpha 32, lr
1e-4, 512px), a VRAM row, and a resolution-must-be-a-multiple-of-32 gate.
* Studio: record the measured LTX-2 training VRAM
Loop peak 11.2 GB, whole-run peak 34.8 GB on a B200 (nf4, rank 32, 512px stills,
batch 1, gradient checkpointing). The run peak is set by the Gemma3-12B
conditioning stage, not by the training loop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep curated Recommended rows searchable and give them their metadata
Two things went wrong with the curated video models the Recommended list
paints from the catalog rather than from a live Hub listing.
Search dropped them. The Recommended search matched the query against
`recommendedIds`, which filters out every id already on disk because a
downloaded model gets its own On Device row. The unfiltered Recommended
list does not filter that way: it renders the curated seeds and badges the
downloaded ones. So a curated model was visible in the list and unfindable
by typing its name the moment it was downloaded, and only a live listing
row could bring it back, which a repo the listing does not return never
gets. `searchableRecommendedIds` unions the seed ids with the listing ids,
seeds first, deduped case insensitively (the HF cache lowercases repo ids),
so both lists agree on what exists.
Rows rendered bare. Everything past the id came from the listing alone, so
a curated row the listing never returns showed no parameter chip and no
capability glyph while its neighbours showed both. The catalog now carries
the two facts nothing else can supply: `totalParams` per artifact and
`capabilities` per group, read through `curatedTotalParamsFor` and
`curatedCapabilitiesFor`. Both are fallbacks only. A listing row wins
wherever there is one, because real tags and a Hub-reported total outrank
anything hand written here.
The parameter counts are measured, not guessed. The MiniMax-H3 figure is
the sum of the tensor shapes in the BF16 GGUF of the FL2VA denoiser that
repo publishes; the LTX-2.3 figure is what the Hub reports for its repo,
carried so the row looks the same offline or rate limited.
One more inconsistency fell out of the same place: `searchRowFits` hides
anything it cannot size (`requireKnown`), and it could not size a curated
repo with no listing row and no "<n>B" token in its id, so turning on
"Fits on device" hid from search a row the unfiltered list still showed.
It now falls back to the curated total the same way it already fell back
to the curated size.
Covered by studio/frontend/tests/recommended-curated-row-metadata.test.ts:
15 assertions over the search pool, the two catalog lookups, the fit check,
and the four picker call sites that read them.
* Load MiniMax-H3 from a hosted pre-quantized denoiser
Video families had no way to reach a hosted pre-quantized DENOISER. VideoFamily
carried gguf_repo and te_prequant_repos (the text encoder) only, and nothing in
video.py consulted a denoiser table, so a MiniMax-H3 load either pulled the full
66.3 GB bfloat16 DiT or nothing at all. The image side already solves this with
DiffusionFamily.prequant_repos plus family_prequant_repo(), so this follows that
shape rather than inventing a second one.
Four parts.
VideoFamily gains prequant_repos, prequant_variant_repos and prequant_subfolder,
resolved by video_family_prequant_repo() / video_family_prequant_schemes(),
mirroring the image resolver. The registries stay separate, as the module header
requires, so the base-id normaliser is local rather than imported.
The shared resolver learns an optional subfolder. The hosted video checkpoints
nest theirs one level down instead of keeping it at the repo root, and the prefix
has to reach BOTH candidate names or the primary 404 is followed by a second one
and the load silently falls back to the dense download. Always a literal forward
slash: these are Hub repo paths, and a Windows join would miss the cache. The
cache and download plumbing already handled a nested name, so only the filename
builder changed, and every existing call site is byte-identical.
The modular workflow builds its denoiser through its own component loader, so
there is no dense module to quantise in place. A hosted checkpoint is therefore
the only way to run that transformer quantized, and pre-seeding it with
update_components() before load_components() is also what stops the dense
download: load_components(names=None) skips a component whose attribute is
already set. Passing names= instead would have forfeited the workflow's own block
pruning, which is what already avoids the 61.7 GB Ref2VA transformer.
Those checkpoints carry the pruned adaLN, where the modulation is a rank-8 affine
factorization of the time-embedding curve plus a shared table, and roughly 40% of
the released model's parameters go. Against the base repo's dense config the
model is four keys short, one over and fifty-one shapes wrong, so the strict load
fails and the checkpoint is unloadable by every route. video_minimax_h3_adaln.py
reshapes the model between from_config and load_state_dict: table lookup with
interpolation instead of the timestep MLP, no SiLU (the table already holds the
activation's own output projected onto the basis), and the modulation cast to the
block stack's dtype, without which the first quantized matmul dies on mismatched
dtypes. Bound per instance, so a dense load in the same process is untouched.
Finally the refusal becomes honest. A single-file pick on a modular family used
to reach the loader only after ~98.7 GB had downloaded AND after the resident
pipeline had been evicted to make room for it, because download-plan returned 200
and validation passed. Both refusals now run in validate_load_request, ahead of
the diffusers availability probe so they still fire where diffusers cannot be
imported, and each names what to pick instead. download-plan forwards
transformer_quant to validation and to the plan, without which the quant-keyed
refusal never fires on the route that stages the download and the plan stages
shards the load never opens.
* Point both MiniMax-H3 schemes at one hosted pre-quantized repo
The two hosted pre-quantized denoisers were split across two repos with the
checkpoint nested one level down, so reaching them needed a mechanism the image
side has never had: a VideoFamily.prequant_subfolder field, a
prequant_subfolder_prefix() normaliser, and a subfolder keyword threaded through
both resolve_prequant_source() and usable_prequant_source().
Both schemes now live in ONE hosted repo, at the root, named <Model>-<SCHEME>.pt.
That is the layout every image-side prequant repo already uses, and it is exactly
what prequant_repo_filename() builds unaided, so the whole mechanism goes. Match
an existing convention and the code should shrink, not grow: -22 lines in
diffusion_prequant.py, -6 in video_families.py, -6 in video.py, with no new
concept to carry.
Landing on the primary name also fixes a memory-planning under-credit.
cached_checkpoint_path() deliberately credits only the PRIMARY filename, so that
a cached legacy artifact cannot pin a stale name once a repo ships the real one.
While these checkpoints were published as transformer_<scheme>.pt the primary
never existed: every hit came through fallback_filename, and planning therefore
read an already-cached checkpoint as "this would have to download" and handed the
pick to GGUF. The primary is now the published name, so the probe hits it.
fallback_filename stays. It still covers repos that have not been renamed, and
dropping it is a separate decision from this one.
Tests: the subfolder-prefix cases are replaced by the naming they now guard --
both schemes resolving to one repo, the primary resolving to a root-level
<Model>-<SCHEME>.pt with no directory component on any platform, the cache probe
being asked for that primary name, and the repo's own scheme suffix being
stripped and replaced rather than carried through. Five mutations run, each
caught by the named test and reverted: two repos again (M1), the primary nested
under prequant/ again (M2), the suffix strip removed (M3), the cache probe keyed
on the fallback (M4), the fallback name dropped (M5).
* Add MiniMax-H3 image and reference video conditioning
* Improve H3 finalization progress
* Report real sd.cpp progress on the Video page instead of a frozen 0 of 30
A native (GGUF) video generation reported phase "denoise", step 0/30 for its
entire run and then flipped straight to "completed". Two separate things were
wrong, and the progress endpoint could not move until both were fixed.
The parser looked for r"(?:step|sampling)\D+(\d+)/(\d+)". sd-cli's sampling
bar contains neither word. It prints
|=========> | 7/30 - 21.50s/it
so nothing ever matched. Anchor the pattern on the bar and on the trailing
speed unit instead. A bare "n/m" is deliberately not enough: an unrelated
ratio in some other log line must not drive the progress bar.
The reader also delivered every redraw one step late. sd-cli redraws in
place, and its carriage return LEADS the record while the newline only
arrives on the final step, so a reader keyed on CR/LF cannot produce step 1
until step 2 has been flushed. Treat the erase-to-end-of-line that closes
each redraw as a terminator too, and read the pipe with buffer.read1 so a
record that carries no newline is not stuck behind a blocking readline.
Escapes are stripped before a record reaches on_log or the error tail.
Streams without a raw .buffer keep the old line iteration.
The same bar shape is printed by three different things, so the parser now
tells them apart rather than reporting whichever came last. Weight load
prints it with a byte rate, and tiled VAE decode prints an identical s/it bar
counting TILES: without a guard a run finished sampling at 30/30 and then
jumped backwards to "step 1/16". Load and decode are real work with no
sampling step, so they report their own phase and a null step rather than a
fake 0 of 30. ETA is measured from the first step, not from job start, so the
one-off weight load is not charged to every remaining step.
Verified end to end against a real CUDA sd-cli MiniMax-H3 generation: the
step advances 1..6 over 6.2s..9.7s of wall clock, load and decode are
reported as themselves, and the ETA tracks.
* Read sd-cli's in-place progress redraws so the Video bar moves during sampling
The native H3 progress bar had two independent causes and fixing either alone
changed nothing observable. The bar pattern is now correct, but the reader still
was not.
sd-cli redraws its sampling bar in place: one printf per step shaped
"\r<bar> <n>/<total> - <speed>\033[K", with a newline only on the final step of a
phase. The drain loop did `for raw in proc.stdout`, which terminates on LF only,
so every redraw sat in the buffer until the next one's carriage return arrived
and the last one until sampling was already over. The Video page saw nothing.
Split the raw pipe into records on CR, LF, or the trailing erase-to-end-of-line,
reading through buffer.read1 with an incremental UTF-8 decoder so a multi-byte
character straddling two reads survives, and strip the CSI escapes before the
record reaches on_log or the error tail. Streams without a raw .buffer (test
doubles, non-pipes) fall back to line iteration.
The new backend test drives the real byte stream through both halves, one flush
per read, and asserts each step is visible on the read that carried it rather
than one redraw later.
* Give MiniMax-H3 first and last frame conditioning in the video backend
MiniMax-H3's released transformer is the FL2VA one: text-to-video is the same
checkpoint run with no keyframes. Studio only ever ran it text-only, so the
Video generate request had nowhere to attach a reference frame.
The load used to prune the block graph to t2va. That argument prunes STATICALLY,
so an fl2va-pruned pipeline runs the keyframe blocks on every request and cannot
serve a text-only one at all: it raises packing an empty conditioning list.
The load now keeps the whole auto graph, which selects per request, and bounds
load_components to the keyframe workflow's component set instead, so the 61.7 GB
Ref2VA partition is no more loaded than before. Measured against the released
checkpoint: a text-only request through this pipeline is bit-identical, video
and audio, to the same request through a t2va-pruned one.
A keyframe is a geometry anchor, not just conditioning, so the canvas comes from
its aspect ratio through the released arithmetic (768 short edge, area capped at
768x1344, both axes rounded to 32) rather than from whatever resolution preset
was selected. An arbitrary size produces a garbled clip rather than an error.
sd.cpp already implements the same conditioning, so that side is the existing
--init-img / --end-img flags with the frames staged as PNGs.
Only MiniMax-H3 declares the capability, and status reports it, so Wan and LTX
do not grow a control that does nothing.
* Cover the MiniMax-H3 keyframe path with tests
Registry (which families declare it, and the canvas a keyframe resolves to),
the load wiring (whole block graph, component set still bounded, VAE encoder
kept), request handling (decode, refusal, canvas override, what reaches the
pipeline call) and the sd-cli argv.
* Add the reference-frame controls to the Video page
First and last frame pickers, shown only for a family whose status declares
keyframe conditioning, so Wan and LTX are unchanged. The Images page's source
picker moves to a shared component rather than the Video page growing a second
upload path; both send the same data URL to the same backend decoder.
While a frame is attached the Resolution preset is disabled and says why: the
frame's aspect ratio resolves the clip's size, the way the model itself does.
The gallery recipe records which ends were pinned.
* Check the keyframe canvas against the pipeline's own resolver
The canvas rule is a checkpoint contract, so pin it to the released
implementation rather than only to hand-written expectations. Skipped where
diffusers does not ship MiniMax-H3, which is most runners.
* Revert the standalone H3 keyframe implementation
oobabooga/unsloth#121 covers first frame, last frame, first-and-last, Ref2VA and
the canvas rule, and it reached the same load construction independently. Two
implementations of the same feature on one branch is worse than either, so this
takes mine back out and leaves the branch ready for that work to land whole.
The one finding worth keeping from it is already reflected there: passing
workflow= to ModularPipeline.from_pretrained prunes the block graph statically,
so the pipeline must be built unpruned and only load_components bounded.
* Keep the pre-quantized MiniMax-H3 denoiser resident so a generation can run
Loading H3 with a hosted pre-quantized denoiser worked, but every generation died
on its first denoise step:
Attempted to set the storage of a tensor on device "cuda:0" to a storage on
different device "cpu". This is no longer allowed; the devices must match.
ComponentsManager.enable_auto_cpu_offload parks every component on the CPU and
moves each one onto the accelerator inside its own pre_forward, that is from
within the block that is already executing. The text encoder and the VAEs survive
that; a torchao-quantized denoiser does not, because the device change reaches
return_and_correct_aliasing, which tries to alias a CPU storage to an accelerator
tensor. Moving the same module at load time, outside any executing block, works.
So place it once at load and take it out of the offload rotation: drop its hook,
unlist it from the other components' eviction candidates, and move it. Everything
else is unchanged, and the encoder and VAEs still offload around it. Keeping it
resident is what asking for a quantized denoiser buys in the first place: the
hosted checkpoint is about 20 GB against 66.3 GB dense.
Verified end to end on a B200: MiniMaxAI/MiniMax-H3 loaded with the hosted fp8
denoiser, then a 1280x768, 124-frame clip generated from a start frame in 167s.
The clip's first frame matches the supplied image and the motion is coherent.
* Apply the pinned Diffusers revision on a fresh install, not just an update
MiniMax-H3 needs a Diffusers revision newer than any published release, and
Studio refuses to load it otherwise. The pin was in
studio/backend/requirements/base.txt, and a clean install.sh run still ended up
on diffusers 0.39.0 from PyPI, every time, with nothing in the log to say so.
base.txt is never installed by install.sh. install.sh installs unsloth itself,
whose own metadata pulls a diffusers release in transitively, and then runs
install_python_stack.py with SKIP_STUDIO_BASE=1, where the base-packages step is
a bare `pass`. So the pin applied on `unsloth studio update` and on the no-torch
path (install.sh installs no-torch-runtime.txt directly) and was dead on exactly
the path a new user takes. Reproduced on a clean install into a throwaway prefix
before and after: 0.39.0, then 0.40.0.dev0 with MiniMaxH3Transformer3DModel
present.
The revision now lives in its own diffusers-pin.txt, installed by a step that
sits outside every skip_base / NO_TORCH branch and after every other
requirements file, so nothing left in the run can re-resolve diffusers back to a
release behind it. No forced reinstall is needed: a direct URL requirement is not
satisfied by a resident registry install, so the step is a no-op once the
environment is already on the pin.
tests/studio/install/test_diffusers_pin.py holds the shape in place: exactly one
requirements file may name diffusers, the pin must be a full commit sha rather
than a moving ref, the install step must sit at function top level rather than
under a conditional, and it must come after every other requirements install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let the Video and Images pickers see a local diffusers pipeline
A model already on disk never reached the pickers' On Device list unless it
happened to keep a weight file beside a root config.json. Every image and video
model downloaded as a pipeline keeps its weights in component subdirs under a
root model_index.json instead, so the hub inventory scan behind /api/hub/local
rejected it. Pointing Custom Folders at one was worse than empty: the LM Studio
publisher walk descended into the pipeline and offered vae, transformer,
text_encoder and audio_vae as four separate models, none of them loadable.
Teach the hub scanner the same pipeline-root test routes/models.py already
applies, in the three places that only accepted a root config plus loose
weights, and keep a pipeline row through the custom-folder format filter: the
layout has no loose weight to classify, so the row is "unknown" by construction
rather than by fault.
* Cover the local diffusers pipeline scan with tests
* Pin what the pipeline exemption must not let through
Three gaps in the cover added with the scan change, each found by mutating the fix and
watching the suite stay green.
The custom-folder format filter now waves a row through on its shape, and nothing said
what it still has to reject: replacing the whole predicate with True passed. A folder
holding a config.json and no weights, which an aborted download leaves behind, reports
the same "unknown" format and no loader can start it, so it pins the boundary.
The predicate is applied to every row the filter did not already accept, and a row's
path can be a GGUF file rather than a directory. A missing path, a file, and a
directory whose model_index.json is itself a directory must answer False rather than
raise, because an exception there fails the scan and empties the picker.
The publisher walk was only covered one level up. Adding the model folder itself as a
scan folder is the obvious thing to do and used to publish vae, transformer and
text_encoder as three models.
* Studio: stop an LTX-2 run publishing into the image LoRA catalog, and refuse a doomed video base before the GPU teardown
Three defects in the LTX-2 video LoRA training path.
A completed video run copied its adapter into loras/diffusion and reported the
copy as catalog_path. That directory is scanned only by the Images LoRA picker,
and core/inference/video.py has no LoRA surface at all, so the mirror was a
multi-gigabyte file nothing could load behind a "Deploy to Create" button that
could not work. _publish_to_lora_catalog now returns None for a video family, so
catalog_path stays unset and the run reports lora_path (and ema_path), which is
the adapter a caller loads directly. The Train panel drops the Deploy button and
the "added to the LoRA picker" claim when there is no catalog path. A LoRA
catalog and a load path for the Video tab is separate, larger work.
LTX2Pipeline landed in diffusers 0.39 and the packaging deliberately keeps an
older diffusers installable on the Python 3.9 hosts this project still supports,
so a supported install can have no LTX2Pipeline at all. Both inference paths
already assert the pipeline class exists before a load; the training preflight
did not, so ltx-2 resolved as trainable, /diffusion/start unloaded the resident
GPU workloads, and only the spawned child failed. resolve_trainable_family now
runs the same assert, which puts it ahead of every teardown in the start route.
That assert also had to stop raising anything but its own ValueError. diffusers
exposes its pipelines through a lazy module, so hasattr is what imports the
submodule, and an install whose own dependencies are unsatisfiable raises
RuntimeError from there. hasattr only absorbs AttributeError, so a partially
usable diffusers escaped the guard exactly the way a missing one used to, as a
bare 500 with the message lost. The attribute probe moves inside the try.
Finally, unsloth/LTX-2-FP8 resolved as a training base. It holds pre-cast
component archives with no model_index.json, no VAE and no pipeline, but it
carries the ltx-2 token so the detector claimed it, the unsloth/* trust gate
passed it, and the gated-access probe ignores the resulting model_index.json 404
because a 404 is not an access problem. The run therefore evicted the resident
models and only then failed inside from_pretrained. The preflight now refuses any
repo the registries list only as a component source, read from te_prequant_repos
across both the image and the video registry rather than by repo id. The rule is
self-limiting: the image FP8 repos are also listed in prequant_repos as
full-model mirrors, so they are bases and resolve exactly as before.
Tests: the LTX-2 routing test no longer claims the component repo is a base; new
tests cover the unpublished video adapter, the refusal on a diffusers without
LTX2Pipeline, the component-repo refusal, and the registry-driven rule sparing
the image mirrors. The two start-route tests assert the 400 arrives with the GPU
teardown never called and the training slot never reserved.
* Studio: pin the pipeline gate's silence on a diffusers whose lazy submodule cannot import
The attribute probe is what imports the pipeline's submodule, so a partially
usable diffusers raises RuntimeError there rather than reporting the class
missing, and hasattr absorbs only AttributeError. Cover it at the guard itself
and at the training preflight that now calls it, where the escape would turn an
actionable 400 into a bare 500.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: MiniMax-H3 joint video and audio LoRA training
Adds the clip dataset layer and the H3 trainer loop.
* Studio: route MiniMax-H3 to its own trainer and cover it with tests
* Studio: match the MiniMax-H3 LoRA targets with a regex and refuse unsupported runs up front
* Studio: sharpen the MiniMax-H3 canvas, timestep-order and trust assertions
* Studio: widen the bf16-only invariant to families whose trainer is not the DiT one
* Scope the paired-axes canvas rule to keyframe requests
A half-specified width/height is long-standing, documented API behaviour: the backend
resolves the missing axis from the family's default preset, in both
validate_video_request_shape and _resolve_keyframes. Applying the new paired-axes rule
as an unconditional request validator rejected those calls with a 422 before family
validation ever ran, breaking existing LTX, Wan, Hunyuan and prompt-only H3 clients.
The rule still holds where it means something: with a keyframe present the canvas is
matched to the source aspect whenever either axis is missing, so the axis the caller
sent would be silently discarded.
* Give MiniMax-H3 its released video flow shift by default
An omitted flow_shift normalized to the identity 1.0 for every family outside
AUTO_FLOW_SHIFT_FAMILIES, and run_h3_lora_training only falls back to _H3_VIDEO_SHIFT
(12.0, from the released scheduler config) when the value is not a number. 1.0 is a
number, so it won: every default H3 run trained against an unshifted video-noise
distribution the sampler never visits, silently and at full cost. An explicit
flow_shift still overrides it.
Also fixes two tests this PR left red against main:
- test_a_non_modular_video_family_is_unaffected asserted a family name that cannot
occur ('ltx-2.3' is an alias of the ltx-2 family), hiding what it is really about;
it now asserts the family it lands on declares no modular workflow.
- the family sweep in test_plan_is_unchanged_when_no_text_encoder_quant_is_requested
now skips a family whose pipeline class this diffusers lacks, the same rule the
listing routes apply, so it stays about the budget rather than about the runner.
* Preflight a MiniMax-H3 dataset with the discovery its trainer runs
/diffusion/start preflights the dataset before freeing the resident GPU models, which
only works while it runs the same discovery as the trainer. It ran the image discovery
unconditionally, so a MiniMax-H3 dataset -- captioned clips, the only thing
run_h3_lora_training accepts -- was rejected at the route with "No captioned images
found" and the trainer could not be reached through the API at all.
discover_training_pairs dispatches on the resolved family. LTX-2 stays on the image
discovery: it trains a style LoRA from stills, so the split is by what the trainer
reads rather than by output modality.
* Judge a frame count against the family's range, not only its lattice
The shape gate refused off-lattice counts but read min_num_frames / max_num_frames
only when composing the error text, while snap_num_frames also clamps to them. A count
can therefore sit exactly on the lattice and still fall outside the range -- MiniMax-H3
is 17k + 5 over 124..345, so 107 and 362 both do -- and the request was accepted and
then rendered at 124 or 345 instead. Accepting one recipe and drawing another is what
this gate exists to prevent.
Inert for every pre-existing family: they declare min 1 and no max, so the request
model's own ceiling remains the only upper bound.
* Treat a hosted pre-quantized denoiser as a component, not a training base
The two registries mean opposite things by prequant_repos. An image family's entry is a
full quantized pipeline mirror, so it is a base. A video family's is the pre-quantized
denoiser alone, which is a component in exactly the way a pre-cast text encoder is: no
model_index.json, no VAE, no scheduler. _component_only_repos read both as bases, so the
H3 denoiser repo passed the training preflight -- its name resolves to the family and
the unsloth/* trust gate accepts it -- and the run evicted the resident GPU workloads
before failing inside from_pretrained.
It is now refused there, naming the base to train from instead.
* Refuse the checkpoint knobs the H3 loop does not implement
run_h3_lora_training neither writes a resume bundle nor restores one, while the shared
DiffusionLoraConfig carries save_steps and resume_from_checkpoint for every family. Both
were accepted and ignored: a caller handing over a resume bundle got a fresh
optimization that then overwrote the outputs it was meant to continue, and periodic
saves silently never happened, each discovered only after an expensive run.
They are now refused in config validation, before the route evicts anything, with a
message that says why. Every other family keeps its checkpointing untouched.
* Recognize a modular pipeline index in both local scanners
A Modular Diffusers pipeline root carries modular_model_index.json and no
model_index.json, and the video loader accepts exactly that pair. Both local scanners
tested only for the conventional index, so such a root was omitted from the Images and
Video pickers and the publisher walk descended into it and offered its component
directories as separate models that no loader can start.
Both scanners now accept either index; their docstrings already promised they were the
same test.
* Record the build recipe on a native H3 result
The native generation result omitted model_kind, gguf_filename, transformer_quant,
text_encoder_quant, memory_mode and offload_policy, all of which sit on the engaged
state and all of which the diffusers path already records. _run_generate persists them
with result.get(...), so each became null in the gallery sidecar and clips generated
from different GGUF quantizations of the same repo could no longer be told apart or
reproduced from their saved recipe.
* Recognize a standalone H3 denoiser GGUF by its filename
MiniMax-H3's GGUFs carry no metadata keys at all, so general.architecture is absent and
the filename is the only evidence there is. The listing predicate keyed solely on the
two bundle repo ids, so a denoiser copied into a custom local directory produced a null
task and was dropped from the Video On Device picker even though the loader validates
exactly these prefixes. The listing now uses the same evidence the load path does.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stage the MiniMax-H3 denoiser partition the requested task loads
The base repo ships transformer/ (fl2va) and transformer_ref/ (ref2va) at 66.28 GB
each and the modular workflow builds exactly one of them, so the scoped file list is
keyed on h3_task and substitutes one prefix for the other. /video/download-plan
forwards the task for the same reason: a References pick used to stage the keyframe
denoiser and then fetch the reference one inline, outside the download manager's disk
preflight, progress and cancellation.
Also normalise guidance on a guidance-free family. MiniMax-H3 declares
supports_cfg=False, so the Diffusers branch forwards no CFG kwarg and the native
branch pins --cfg-scale 1.0; keeping the requested number in the result labelled the
clip and its gallery sidecar with a value that reached no sampler.
* Refuse a quantized MiniMax-H3 reference pick on the download-plan route too
The hosted prequants are keyframe (fl2va) denoisers, and validate_load_request
already refuses pairing one with a ref2va load. The plan route called that
validation without the task, so the same request that /video/load answers with a
400 came back as a 200 plan staging a checkpoint the load would never open.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard, record and bound what the MiniMax-H3 trainer actually does
Four fixes from the training review:
int8 training now passes the family to _int8_quantize_base, and that helper applies
the family's small-M padding as well as its exclusions. Without the family H3's
adaln_proj -- Linear(2688 -> 96768) on the dense checkpoint, so above the 512-feature
floor -- was quantized and then raised on the first step at M = 1, after the whole
66.3 GB base had loaded; context_embedder and the token_refiner blocks are absent from
both exclusion lists precisely because they are meant to be padded instead.
The adapter export writes the PEFT config as safetensors metadata in diffusers' own
layout, so a non-default lora_alpha survives the round trip. load_lora_adapter reads
the rank off the B matrices and sets lora_alpha = r when the file carries no metadata,
so rank 16 / alpha 32 came back at half its trained strength.
A clip whose soundtrack is materially shorter than its video is refused instead of
zero-padded to the full window. The mandatory-audio check only asks whether the
container declares a stream, so an almost-silent one passed it and then trained the
shared adapter on the silence that check exists to keep out; the few-millisecond
container tail the comment describes is still allowed.
center_crop and random_flip are normalised to what the clip path does (one centre
cover-crop, no flip). Both schema defaults said the opposite, so every run recorded
augmentation that never happened.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refuse or normalise the remaining H3 knobs the loop does not implement
Four more of the same class, judged by the same rule the trainer already applies to
cfg_dropout, weighting_scheme and batch size: a setting that is accepted and then
dropped makes the recorded run describe a recipe that did not happen.
Refused, because the value can only be an explicit non-default: mixed_precision other
than bf16 (the loop hard-codes the bf16 weight dtype, autocast and torch_dtype, so 'no'
ran bf16 while recording full precision), compile_transformer='on' (never invoked -- the
packed sequence changes length with every caption, and the run already reports
compiled=False), and cond_cache_dir (no conditioning cache exists here, so the directory
promised a saving that never happened).
Normalised, because the SCHEMA DEFAULT is the one the loop disagrees with and refusing
would 422 every untouched request: snr_gamma, which defaults to 5.0 while the step is a
plain unweighted MSE, alongside the two augmentation knobs.
* Make the MiniMax-H3 trainer reachable from the Train panel
family_train_infos keyed its precision branch on _DIT_TRAIN_FAMILIES while H3 is in
_FLOW_TRAIN_FAMILIES, so /diffusion/info reported precision_modes: [] for it. The panel
reads an empty list on a non-SDXL family as 'this GPU cannot train this family' and
disables Start with 'Not supported on this GPU', which made the trainer this branch adds
unreachable from Studio on every host. The flow families are exactly the ones that
expose base_precision and need bf16 on CUDA, which is what that branch decides, so it
reads the flow set now. SDXL is in neither and keeps its mixed_precision lever.
fp8 and mxfp8 are declared denied for H3 training, matching the refusal the trainer
already raises: its packed sequence runs three modalities through one set of linears, so
the activation range is not the per-family range the fp8 filter was measured against.
supports_compile stays DiT-only, since compile_transformer='on' is refused here -- the
packed sequence changes length with every caption, so each step would re-trace.
* Answer the MiniMax-H3 restrictions before the start route evicts anything
The H3 trainer's config-only refusals ran in the worker, so a batch > 1, a non-bf16
precision, a weighting scheme, a compile request or a conditioning-cache directory
reached it only after the start route had freed the user's resident GPU models for a
run that then 400'd. They move into h3_train_unsupported_reason, which the route call…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A blanket `--type` currently quantizes 1-D weights whenever their length happens to divide the block size. A 1-D weight is a per-channel gain (LayerNorm/RMSNorm), never a matmul weight, so every channel of a block shares one scale and one min while a gain vector has no reason to be locally smooth.
Until now these survived only by accident: FLUX q_norm/k_norm are `[128]` and `128 % 256 != 0`, so the existing block-size check spared them. MiniMax-H3 norms are `[5376]` and `5376 % 256 == 0`, so a blanket `--type q4_K` crushed 106 norm scales to 4 bits.
The failure is quiet, which is what makes it worth guarding. The model still loads and still renders a plausible-looking video, so a "does it run" check passes. Measured against a bf16 render of the same prompt and seed:
An LPIPS of 0.981 is essentially uncorrelated with the reference.
The rule sits beside the existing `.bias` and `.scale` exclusions and costs almost nothing in size: on H3 the 106 spared tensors are 5376 floats each.