Tested Spectrum, Block Cache, SageAttention2, and ConvRot GGML implementations that may be useful for SD.cpp

[0xShug0]

Hi @leejet,

I'm the maintainer of audio.cpp. I recently implemented the MiniMax-H3 text-to-audio path in GGML (released under the dev branch). The DiT path is effectively shared with video generation.

During the implementation, I tested several DiT optimizations that may also be useful for SD.cpp:

Spectrum-style acceleration
First-block / BlockCache
SageAttention
INT8 ConvRot

The biggest gain came from Spectrum. For a 30s H3 audio at 20 steps, Spectrum improved RTF from about 0.56 to 0.30. BlockCache also provided meaningful speedups, although quality was more sensitive to the number of steps. INT8 ConvRot gave consistent speedup on the normal DiT path, around 9%, and the optimizations could be combined.

I'm not planning to spend much time supporting or optimizing video model pipelines in audio.cpp. H3 video support mostly comes "for free" because the model produces audio and video latents jointly. These GGML implementations may be useful for video diffusion workloads in SD.cpp.

On RTX 5090, for the 1344x768 / 124 frames (5 seconds) / 20 steps case from the official comfyui workflow (the "Realistic live-action cinematic look, action movie trailer..." one) :

INT8 ConvRot, no spectrum --- 260.839 s
INT8 ConvRot + spectrum --- 114.035 s

For video, INT8 ConvRot + spectrum actually looks better.

Feel free to use the implementations at https://github.com/0xShug0/audio.cpp/tree/dev if they’re useful.

[0xShug0]

BTW, there are also some benchmarks that may be useful for understanding the trade-offs, although they’re audio-only.
https://github.com/0xShug0/audio.cpp/blob/dev/docs/reports/minimax_h3_performance.md

[Green-Sky]

Spectrum caching, among others, is already supported.

Also i think conv rot was merged recently too.

[stduhpf]

Also i think conv rot was merged recently too.

If you're talking about #1857, that's still pending, not merged yet.

[0xShug0]

Ah never mind. I thought SD.cpp pinned a specific version of upstream GGML.

[leejet]

Thanks for sharing! Spectrum's caching approach has actually been supported in sd.cpp for quite a while. I also added CUDA support for int8_convrot about a week ago.

Over the past few days, I've been working on adding Vulkan support for int8_convrot, so I held off on merging the changes until that was done. I've merged everything now.

If you haven't implemented Vulkan support for int8_convrot yet and are interested in adding it to audio.cpp, feel free to use the relevant code.

[0xShug0]

@leejet I checked your changes and found that the way you extended GGML is quite different from mine 😄. Your approach is definitely cleaner architecturally, but it’s a bit overkill for audio.cpp’s use case, especially since only one model benefits from it. In general, for TTS/Speech audio generation, I’ve found INT8 ConvRot to be less accurate than Q4. I’ll probably keep INT8 ConvRot support CUDA-only as an experimental feature for now.

I may be missing something, but how is SageAttention2 implemented in sd.cpp?

Also, now that I know you use a customized GGML, I’m wondering whether some of the localized GGML optimizations in sd.cpp could benefit DiT-based TTS models as well. Would you like to share any of the relevant changes or PRs?