rtx-3090

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

First public benchmark of llama.cpp speculative decoding on Qwen3.6-35B-A3B with a single RTX 3090 (post PR #19493 merge, 2026-04-19). 19 configurations covering ngram-cache, ngram-mod, and classic draft with vocab-matched Qwen3.5-0.8B. Finding: no variant achieves net speedup on Ampere + A3B MoE. Raw JSON, plots, full reproducibility.

The bare metal in my basement

Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig

Local LLM speed measurements from a 4x RTX 3090 rig, with exact launch commands and per-run provenance.

Qwen3.6 vLLM Toolkit — Launcher + Templates, Optimized for 2×24 GB. vLLM launcher with an optimized hybrid chat template drawing from the best community fixes for Qwen 3.6 27B.

DeepSeek-V4-Flash-0731 on 8x RTX 3090 (SM86) with vLLM — verified FP8 serving, benchmarks, build guide, and reproducible release.

Local agentic coding stack: Hermes Agent + Qwen3.5-27B + GLM-4.7-Flash on dual RTX 3090s. Daily-driver agentic work, no cloud, no metering. Companion to blog.zacharycangemi.com.

Dual RTX 3090 local LLM infrastructure — 48 GB VRAM AI cluster build. Hardware spec, photos, power-limit script. Companion to the blog series at blog.zacharycangemi.com.

Reproducible vLLM recipe for shawnw3i/Huihui-Qwen3.6-27B-abliterated-AWQ-MTP on 2× RTX 3090 in a Proxmox LXC. MTP n=3, 256K context, full vision+tool-calling+reasoning. Silent 24/7 operation at 250W per card. Companion to the base-model recipe.

poolside Laguna-S-2.1 INT4 + DFlash speculative decoding on 4x RTX 3090: 200K context, 282 tok/s peak decode, gate-proven with a 190K-token prompt. Full levers menu + failure catalog.

Reproducible vLLM recipe for Qwen3.6-27B (AWQ-BF16-INT4) on 2× RTX 3090 in a Proxmox LXC. MTP, 256K context, full vision+tool-calling+reasoning. Bare-LXC alternative to Dzombak's Docker recipe. Benchmarks, gotchas, GPU passthrough config included.

2.28× faster Claude Code on a local Qwen3.6-27B int4 (RTX 3090) — turbo-64k + long-100k profiles, MTP, tool calling, corruption guards.

Veizik — hardware-aware local AI media runtime. Hardware detection, low-memory execution planning, local licensing and experimental AI-video rendering.

One-click deployment of Qwen3.8-27B on a single RTX 3090 (24GB) on Windows 11: Q4_K_XL quantization + MTP speculative decoding + 96K context window, exposed as an OpenAI-compatible llama-server, averaging ~47 tok/s with ~1.5× lossless MTP speedup

Prebuilt Linux sm_86 binaries for the NInfer RTX 3090 engine (upstream ships Windows only) + a bash CLI that rents the 3090 on Vast.ai and pays $0.00 when idle.

100% local voice assistant with Tool Calling, neural TTS, and streaming responses. Runs on RTX 3090 with Ollama + Kokoro TTS + FastAPI. Privacy-first AI.

Run a local, uncensored coding model as an autonomous security-research agent on a single GPU. Tuning findings, OpenCode setup, and self-contained pentest labs.