LLM Hardware Calculator

Up to 262144 for qwen3-*. Quality degrades beyond native.

qwen3-32b · Q4_K_M · KV=Q8

weights ~17.6 GB · KV 131 KB/token · need ~36.8 GB for 131,072

Bottleneck: KV 47% / weights 48% / overhead 5%

  • 2x RTX 3090max 131,07243-64 tok/s
  • RTX 4090 48GBmax 131,07229-43 tok/s

Units: decimal GB=1e9 bytes, KB=1e3 bytes. Never GiB/KiB.

CPU offload is not a v1 fit mode; mmap affects load time only, not VRAM accounting.

Decode-only estimate; prefill (large context reads) not modeled — on MoE giants prefill is the real bottleneck.

decode-only estimate; anchors: llama.cpp single stream, 2026-09

About these estimates

  • Single-stream decode (batch=1) calibrated to llama.cpp measurements, 2026-09.
  • Decimal units: GB=1e9 bytes. Unified machines reserve 4 GB for the OS.
  • MoE decode reads active experts only, discounted 0.35× (latency-bound).
  • Q8 KV halves cache memory with negligible quality loss.
Which GPU or Mac fits your model, quant, and context — and at what tok/s. Offline decode estimates from llama.cpp anchors (decimal GB).