Total GPU Memory = Model Static Weights + Dynamic Context KV Cache + Forward Decoding Activation Buffers + OS & CUDA Runtime Overhead
Static Weights VRAM
1.05x Overhead
VRAM_Weights = N_params * Bytes_Per_Weight * 1.05
The 1.05 multiplier accounts for unquantized embedding tensors, classification lm_head, and tokenizer vocabulary tables.
FP16: 2.00 B
Q8_0: 1.08 B
Q4_K_M: 0.58 B
Dynamic KV Cache (MLA vs GQA)
1.06x PagedAttn
GQA: 2 * L * N_kv * D_head * Bytes * Ctx * 1.06
MLA: L * D_latent(576) * Bytes * Ctx * 1.06
DeepSeek Multi-head Latent Attention (MLA) compresses the KV state into a 576-dim latent vector, reducing memory consumption by up to 82% compared to standard GQA.
Batch=1 autoregressive decoding is memory-bandwidth bound. 0.68 represents realistic hardware memory-controller efficiency during decoding.
Hardware Architecture FAQ & Buying Guide
Even lightweight 7B or 8B models (like Llama 3.1 8B or Qwen 2.5 7B) require ~4.8 GB for Q4 weights alone. When you add Windows desktop DWM overhead (0.8 GB), PyTorch/CUDA buffers (0.5 GB), and run an 8k-16k context window, KV Cache consumes another 2-3 GB. This immediately breaches 8.0 GB, causing sudden CUDA OOM crashes or catastrophic PCIe throttling down to ~1 tok/s.
Attention Architecture KV Cache Scaling Benchmark (GQA vs DeepSeek MLA)
KV cache footprint is the primary cause of sudden CUDA Out-of-Memory errors during long-context sessions. While standard Grouped-Query Attention (GQA) stores full uncompressed key and value vectors per token across all layers, DeepSeek Multi-Head Latent Attention (MLA) projects the joint KV state into a compact 576-dimensional latent space, reducing memory consumption by up to 82% at 32k to 128k context windows:
Context Window
Traditional MHA (FP16)
Standard GQA (FP16)
DeepSeek MLA (FP16)
MLA Memory Savings
Min Safe VRAM Headroom
4,096 Tokens (4k)
4.10 GB
1.03 GB
0.29 GB
-72% VRAM
8 GB GPU (Entry)
8,192 Tokens (8k)
8.20 GB
2.05 GB
0.58 GB
-72% VRAM
12 GB GPU (RTX 3060)
16,384 Tokens (16k)
16.40 GB
4.10 GB
1.16 GB
-72% VRAM
16 GB GPU (4070 Ti S)
32,768 Tokens (32k)
32.80 GB
8.20 GB
2.32 GB
-72% VRAM
24 GB GPU (RTX 4090)
65,536 Tokens (64k)
65.60 GB
16.40 GB
4.64 GB
-72% VRAM
32 GB GPU (RTX 5090)
131,072 Tokens (128k)
131.20 GB
32.80 GB
9.28 GB
-72% VRAM
48GB+ / Apple Silicon
Bus Bandwidth & Layer Offload Throttling Benchmark
Single-user autoregressive token generation is 100% memory-bandwidth bound. When model layers overflow dedicated VRAM, weights must be transferred across the physical system bus for every token generated. Below is our empirical benchmark measuring throughput degradation across PCIe and external interconnects:
Interconnect Architecture
Bus Bandwidth
Relative Bandwidth
Est. Decoding Speed (32B)
Latency Per Token
User Experience
NVIDIA RTX 5090 (GDDR7 VRAM)
1,792 GB/s
100.0% (Baseline)
~62 tok/s
~16 ms
Instant Fluid Real-Time
NVIDIA RTX 4090 (GDDR6X VRAM)
1,008 GB/s
56.2%
~42 tok/s
~24 ms
Fluid Real-Time
Apple M2 Ultra (Unified Memory)
800 GB/s
44.6%
~28 tok/s
~36 ms
Very Fast Interactive
PCIe 5.0 x16 Offload to DDR5
63.0 GB/s
3.5%
~2.6 tok/s
~385 ms
Noticeably Sluggish
PCIe 4.0 x16 Offload to DDR4/5
31.5 GB/s
1.8%
~1.2 tok/s
~830 ms
Severe Bottleneck
Thunderbolt 4 / USB4 eGPU Bus
3.2 GB/s
0.18%
~0.15 tok/s
~6,600 ms
Unusable Slideshow
Frequently Asked Questions on VRAM Math & Architecture
Mathematical proofs, attention scaling laws, and hardware bus equations:
What is the exact formula to calculate local LLM VRAM requirements?
Total VRAM is calculated as: Total_VRAM = Weights_GB + KV_Cache_GB + Activation_GB + CUDA_Overhead_GB. For a Q4_K_M quantized model, Weights_GB ≈ Parameters_B × 0.58 × 1.05. Dynamic KV Cache depends on attention architecture (GQA vs MLA) and scales with context length and batch size. OS and CUDA context runtime overhead typically requires 1.0GB to 1.4GB on Windows/Linux.
How does DeepSeek Multi-Head Latent Attention (MLA) compress KV cache memory?
Standard GQA caches separate uncompressed key and value vectors of dimension d_head per KV head per layer (2 × n_layers × n_kv_heads × d_head × context_len × precision). In contrast, DeepSeek MLA low-rank projects the joint KV state into a single compressed latent vector of dimension d_c = 512 plus decoupled positional RoPE keys d_rope = 64 (576 scalars per token per layer), achieving up to 82% VRAM reduction at 32k to 128k context lengths.
Why does PCIe layer offloading reduce generation speed by over 90%?
During single-batch token generation, each token requires every active layer weight to be fetched. Dedicated GDDR6X/GDDR7 VRAM provides 1,008 to 1,792 GB/s bandwidth. A PCIe 4.0 x16 bus provides only 31.5 GB/s (less than 2% of VRAM bandwidth). When layers are offloaded to system DDR RAM, weight streaming across PCIe creates an insurmountable pipeline stall, dropping generation speed from 30+ tok/s down to ~1.2 tok/s.
What is PagedAttention memory fragmentation in local inference servers?
Traditional contiguous memory allocation causes up to 60% memory waste due to internal and external fragmentation when sequence lengths vary dynamically. PagedAttention (as pioneered in vLLM) partitions the KV cache into fixed-size virtual memory blocks (pages), reducing memory waste to less than 4% and enabling significantly larger batch sizes and context windows in dedicated VRAM.