LLM Hardware Calculator

KV cache

Up to 262144 for qwen3-*. Quality degrades beyond native.

qwen3-32b · Q4_K_M · KV=Q8

weights ~17.6 GB · KV 131 KB/token · need ~36.8 GB for 131,072

Bottleneck: KV 47% / weights 48% / overhead 5%

  • 2x RTX 3090max 131,07243-64 tok/s
  • RTX 4090 48GBmax 131,07229-43 tok/s

Units: decimal 1 GB=109 bytes1\,\text{GB} = 10^{9}\,\text{bytes}, 1 KB=103 bytes1\,\text{KB} = 10^{3}\,\text{bytes}. Never GiB/KiB.

CPU offload is not counted as a fit mode; mmap affects load time only, not VRAM accounting.

Decode-only estimate; prefill (large context reads) not modeled — on MoE giants prefill is the real bottleneck.

decode-only estimate; anchors: llama.cpp single stream, 2026-09

About these estimates

  • Single-stream decode (batch=1\text{batch} = 1) calibrated to llama.cpp measurements, 2026-09.
  • Decimal units: 1 GB=109 bytes1\,\text{GB} = 10^{9}\,\text{bytes}. Unified machines reserve 4 GB for the OS.
  • MoE decode reads active experts only, discounted 0.35× (latency-bound).
  • Q8 KV halves cache memory with negligible quality loss.

About this tool

This calculator estimates whether a large language model fits on a given GPU or Mac and how fast it would generate text. It is aimed at people running models locally with llama.cpp and similar tools. Quantization stores model weights at lower precision, which cuts memory use at a small cost in quality; this calculator counts Q4_K_M as 0.55 bytes per parameter. The KV cache holds attention data for every token in the context.

Two hardware numbers decide the outcome. Memory capacity determines whether the weights and the context you want load at all, while memory bandwidth determines how quickly tokens appear once they do. A card with plenty of memory but low bandwidth, such as an older data-center GPU, can hold a large model and still generate slowly.

How it works

In Find hardware, choose a model, a quantization, and a KV cache precision, then enter the context length and the minimum speed you accept in tokens per second; YaRN is optional. The tool lists machines that pass, with memory needs and speed ranges. In Check my hardware, pick a machine, choose quantizations to compare, and set a target context to see which models fit.

Required memory is the weight size plus the KV cache for the full context plus a fixed 2 GB overhead. Weights in GB equal billions of parameters times bytes per parameter: 0.55 for Q4_K_M, 0.65 for Q5_K_M, 0.75 for Q6_K, 1.07 for Q8_0 and 2.0 for BF16. The cache per token is 2×L×Hkv×dhead2 \times L \times H_{kv} \times d_{head} bytes at Q8 and twice that at FP16, where L is the number of layers. Decode speed is estimated as 50% to 75% of the hardware's effective bandwidth divided by the active weight size, with lower efficiency factors for non-CUDA hardware such as AMD GPUs and Apple chips.

Example

For qwen3-32b at Q4_K_M with a Q8 KV cache, weights take 32×0.55=17.632 \times 0.55 = 17.6 GB and the cache takes 131.072 KB per token. On an RTX 4090 with 24 GB, after 2 GB of overhead and a 0.5 GB reserve, about 29,754 tokens of context fit, so a 32,768-token target misses. Estimated decode speed is 28.6 to 43.0 tokens per second.

Now take llama3.3-70b at Q4_K_M on an M4 Max with 128 GB of unified memory. Weights need 70×0.55=38.570 \times 0.55 = 38.5 GB and the Q8 cache needs 163.84 KB per token, so a 32,768-token context requires about 45.9 GB in total and even the full native 131,072 tokens fits after the 4 GB reserve. The bottleneck is speed: the estimate is only about 5.0 to 7.4 tokens per second, because every token reads all 38.5 GB of weights over 546 GB/s of bandwidth.

Assumptions and limits

Estimates cover single-stream decoding only, calibrated to llama.cpp measurements; prompt processing time is not modeled. Memory uses decimal gigabytes, a fixed 2 GB overhead, and a reserve of 0.5 GB on discrete GPUs or 4 GB on unified-memory machines. CPU offload and partial fits are not counted. The model and hardware lists are fixed catalogs, and real speeds vary with drivers and settings.

Frequently asked questions

How much VRAM do I need to run an LLM locally?

Roughly the size of the quantized weights plus the KV cache for your context length, plus some overhead. A 4-bit quantization needs a little over half a gigabyte per billion parameters for weights alone, so a 32B model needs about 18 GB before context. Long contexts add up quickly, so check the cache figure for your target length before buying hardware.

What does Q4_K_M mean?

Q4_K_M is a llama.cpp GGUF quantization type that stores most weights with about 4 bits using the k-quant method, with some tensors kept at higher precision. It is a common default because it balances file size and output quality. Q5_K_M and Q6_K keep more detail at a larger size, while Q8_0 is close to the original model in quality.

Why is generation speed limited by memory bandwidth?

When generating one token at a time, the hardware must read all active weights from memory for every token. Tokens per second is therefore roughly memory bandwidth divided by the size of the active weights. That is why a smaller quantization runs faster on the same machine and why bandwidth matters more than raw compute for local chat.

Why do mixture-of-experts models need a lot of memory but run fast?

A mixture-of-experts model must keep all experts in memory, but each token uses only a few of them. Memory needs follow total parameters, while decode speed follows the smaller number of active parameters. For example, qwen3-30b-a3b needs memory for 30 billion parameters but reads only about 3 billion per token.

Why do my measured tokens per second differ from the estimate?

The estimate is a range calibrated to single-stream llama.cpp decoding, so other software, drivers, batch sizes, and thermal limits move real numbers up or down. Prompt processing is not modeled, so long prompts add waiting time before the first token appears. If part of the model is offloaded to system RAM, speed drops far below the estimate, because the calculator counts only full fits.

Link with your values

Add these parameters to this page’s address to open it with the values filled in, for example to share a calculation or to link to it from an AI assistant. The result is calculated in the browser: the page HTML shows an example for the default values, and the tool sets the data-url-params attribute once the link’s values are applied.

ParameterValueIf omitted
modeCalculationone of: need, hardwaredefault need
modelModelone of: qwen3-8b, qwen3-14b, qwen3-30b-a3b, qwen3-32b, llama3.1-8b, llama3.3-70b, qwen3-235b-a22b, qwen3-480b-a22b, qwen3-4b, gpt-oss-20b, gpt-oss-120b, deepseek-r1-distill-qwen-32b, qwen3-coder-30b-a3b, phi-4, qwen2.5-coder-32b; used with mode = needdefault qwen3-32b
quantQuantone of: Q4_K_M, Q5_K_M, Q6_K, Q8_0, BF16; used with mode = needdefault Q4_K_M
context_tokensContext (tokens)whole number; allowed ≥ 0; used with mode = needdefault 131072
min_tokens_per_sMin tok/snumber; allowed ≥ 0; used with mode = needdefault 20
hardwareHardwareone of: rtx_3090, rtx_4090, rtx_4090_48gb, rtx_5090, tesla_p40, tesla_p100_16gb, tesla_v100_32gb, mi50_32gb, 2x_rtx_3090, m4_pro, m5_pro, m5_max, m4_max, m3_ultra, dgx_spark_gb10, dgx_station_gb300; used with mode = hardwaredefault rtx_4090
quantsQuantsvalues separated by commas: 2,4,4,6; used with mode = hardwaredefault Q4_K_M,Q8_0
target_context_tokensTarget contextwhole number; allowed ≥ 0; used with mode = hardwaredefault 131072
kv_cacheKV cacheone of: q8, fp16default q8
yarn_tokensYaRN (optional)whole number; allowed 1–262144optional

Examples

  • ?model=llama3.3-70b&quant=Q5_K_M&context_tokens=32768&kv_cache=fp16&min_tokens_per_s=15
  • ?hardware=m4_max&quants=Q4_K_M,Q6_K&target_context_tokens=65536

Write decimals with a point. Encode a space as %20, + as %2B, # as %23 and & as %26.

Result elements: [data-result-block="llm-hardware"]