GPU VRAM calculator for LLMs

How to read this result

Model weights: this is the main item. A model with N billion parameters weighs N × 2 GB in FP16, N GB in INT8, N ÷ 2 GB in INT4. INT4 quantisation therefore divides the memory taken by the weights by 4, with a loss of quality that is generally small.

KV cache: the memory that stores the context of each request, computed from the real architecture of the model: 2 (K and V) × layers × KV heads × head dim. × bytes per element, all multiplied by the context window and the concurrency. The number of KV heads encodes GQA (Mistral 24B: 8 KV heads for 32 query heads, so a cache divided by 4 compared with classic attention). The vLLM FP8 option stores each element on 1 byte instead of 2: the cache is halved with no noticeable impact in production. This item grows with the context length and the number of concurrent requests: it is often what blows up when you serve many users at once.

“My card shows 95% utilisation”: that is normal with vLLM. The engine pre-reserves ~90% of the VRAM at startup (gpu_memory_utilization=0.9) and fills all the free space beyond the weights with the KV cache pool. An A100 80 GB serving Mistral 24B in FP16 therefore looks “saturated” in nvidia-smi even though the real requirement is around 55 GB: the rest is pre-allocated cache to absorb concurrency.

CPU swap & preemption: the KV cache lives in the pool that vLLM pre-allocates on the GPU. When concurrency asks for more cache than that pool can hold, vLLM does not crash: it preempts requests (recomputing their context later) or moves part of the cache into CPU RAM (swap_space). It works, but the time to first token (TTFT) climbs. The tool flags this critical zone as soon as the margin falls below ~10% of the recommended GPU, which is the moment to move up a tier, reduce the context window, or switch the KV cache to FP8.

MoE models (Mixture of Experts): a model such as Mixtral loads all of its experts into VRAM (the weights counted above reflect the total parameter count, not only the active experts), and its working memory is higher. We therefore assume activations at ~18% instead of ~10% for a dense model, which is what the “Architecture type” selector does. On top of that, in every case, comes a fixed baseline of about 3 GB for the CUDA runtime and the vLLM framework: this floor value, independent of model size, avoids underestimating the footprint of a small model (and therefore a risk of OOM at startup).

VRAM lifecycle & daily wipe: vLLM pre-allocates its KV cache pool when the server starts and manages it in pages (PagedAttention), so there is no “memory leak” nibbling away at VRAM request after request. An inference server running for weeks can, however, see its memory fragment. Our operations shut down and reset the processing servers every night: on restart, VRAM allocation begins from zero (KV pool rebuilt from scratch, logs and state purged). The sizing computed here therefore corresponds to a “clean” server every morning, an advantage both for memory stability and for confidentiality (no data persists on the GPU from one day to the next).

Decoding throughput (tokens/s): generation by an LLM is bound by memory bandwidth, because to produce each token the GPU re-reads the entire set of weights. Single-request throughput is therefore estimated as throughput ≈ memory bandwidth ÷ weights, with a real efficiency of 50 to 75% of peak (hence a range). This is an order of magnitude for a single request: the aggregate throughput of vLLM under concurrency (continuous batching) is far higher, and real values depend on the model, the inference kernel and the context length. Not to be taken as a guarantee.

This sizing exercise is exactly what we carry out for every sovereign deployment: choosing the right model, the right quantisation and the right GPU for your real workload. See our LLM models page and the glossary.