SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers.

row-major matmul optimization

An innovative library for efficient LLM inference via low-bit quantization

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.

258 KB WASM runtime for Needle a 26M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.

🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-oriented durability with HNSW + DiskANN indexing for best-in-class performance.

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, deployed and benchmarked. Generates in 16 GB of RAM.

Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.

VKNN — Vulkan Neural Network: dependency-free C++17 on-device inference engine that runs ONNX models entirely on Android GPUs. CNNs, YOLO, LLMs, VLMs, 3D Gaussian Splatting; int4 quantization; compile once to .vxm.

rust library to write integer types of any bit length into a buffer - from `i1` to `i64`.

Low-bit LLM inference engine in Rust + CUDA. Converts models to a 4-bit .wstone format (3.75x smaller) and runs them on consumer GPUs. Built for memory-bandwidth-bound decode on Turing, where bytes per weight — not TOPS — sets token throughput.

Three ComfyUI nodes that show the real runtime precision of MODEL, CLIP, and VAE objects.

Backprop-free learning study: spiking (LIF) neurons + Forward-Forward + JEPA + int4 QAT, with a full ablation notebook.

Low-cost AMD/ROCm stationary-expert fabric for configurable trillion-parameter sparse MoE inference, with packed-INT4 HIP kernels, activation-only RoCE transport, hardware/BOM design, release gates, and reproducible roofline tooling.

Post-training LLM weight compression for low-RAM machines: Q4/Q8 quantization, green-format repair, AVX2 CPU inference, optional CUDA. ~45% less RAM at ~99.9% quality.

SSD-resident INT2/INT4 KV cache with asynchronous staging and fused CUDA dequantizing decode attention for long-context LLM inference.

PyTorch implementation of TRIAD-PTQ (Trace-Router-Interaction-Aware Decomposition) — weight-only INT3/INT4 PTQ for compact LLMs and edge CNNs/ViTs, with real benchmarks on SmolLM/TinyLlama/MobileNetV2/EfficientNet-B0/MobileViT-S.

poolside Laguna-S-2.1 INT4 + DFlash speculative decoding on 4x RTX 3090: 200K context, 282 tok/s peak decode, gate-proven with a 190K-token prompt. Full levers menu + failure catalog.