Why did my model crash with CUDA Out of Memory (OOM) if the weights fit my VRAM?

Model weights are only one part of total VRAM consumption. When prompting a model, the KV Cache (Key-Value Cache) stores all historical tokens. At 32k or 64k context, standard GQA attention requires 4GB to 12GB of additional VRAM. Combined with CUDA context buffers (500MB) and desktop display overhead (0.8GB on Windows), total memory spikes beyond hardware capacity.

What makes DeepSeek V3 / R1 MLA architecture so VRAM-efficient?

DeepSeek Multi-Head Latent Attention (MLA) projects key and value representations into a low-dimensional compressed latent space (latent dimension 512 + decoupled RoPE 64) before caching. Standard GQA must cache all uncompressed KV heads per layer. At 64k context, DeepSeek R1 consumes up to 85% less KV cache VRAM than comparable GQA models like Llama 3.3.

Which quantization format is best for everyday local LLM inference?

Q4_K_M (4-bit Medium) is the universal industry standard sweet spot. It reduces weight memory by 72% compared to FP16 while maintaining over 99% of original model perplexity and reasoning quality.

Can I run models that exceed my local GPU VRAM with CPU offload?

Yes, using Ollama or llama.cpp layer offloading (-ngl), but PCIe 4.0 x16 bandwidth (31.5 GB/s) and DDR5 system RAM bandwidth bottlenecks will cause generation speeds to drop sharply to ~1 to 2 tokens per second.

BENCHMARK SPECIFICATIONLocal AI VRAM & Hardware Telemetry

Can Your Hardware Run This AI Model?

Precise mathematical VRAM calculations, dynamic DeepSeek MLA & GQA KV cache scaling, and prompt prefill speed estimates. Zero guesswork.

89 GPUs

50 Models

MLA Engine

I have this GPU:

I want to run this Model:

Hardware Rig & GPU Configuration

Count:

Popular:

Model Architecture & Specifications

14.7B Params

Popular:

Can Run Perfectly

optimal

Entire model & KV cache fit fully into VRAM with pure GPU acceleration.

Decode TPS

~78.8 tok/s

TTFT Latency

~3476 ms

VRAM Used

13.53 /24G

Weights: 8.95GKV: 3.18GBuffers: 1.4GFree +10.47G

••Report

Quantization Precision:0.58 bpw

Context Window:16k Tokens

Ollama Run Command

ollama run deepseek-r1:14b

vLLM Production Serve

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-14B --gpu-memory-utilization 0.90 --max-model-len 16384

Hardware Diagnostics & Benchmark Suite

Hardware Compatibility Matrix

Browse all 50+ models evaluated against your GPU. Instantly inspect 100% Fit, Tight Fit, and OOM deficits in a CanIUse-style grid.

Open Matrix Directory

Dual-GPU Showdown Arena

Head-to-head battle on the exact same model. Test whether RTX 3060 12GB beats RTX 4060 8GB, or if bandwidth trumps capacity.

Launch Showdown Arena

Best GPU Finder (Reverse Matcher)

Pick your target model. We calculate minimum VRAM requirements and rank budget, sweet-spot, and flagship GPUs.

Find Best GPU

Cluster Scaled Serving

gpuy.dev On-Demand Compute

Fine-Tune or Scale DeepSeek-R1-Distill-Qwen-14B on NVIDIA RTX 4090 (24GB)

Local inference runs well, but LoRA fine-tuning and multi-user vLLM serving require 40GB+ VRAM with FP8/FP16 optimizer states. Spin up dedicated clusters starting from $0.35/hr.

2x Faster Speed

$0.35/hr

Breakeven: ~5,286 hrs vs hardware purchase

Per-second billing

Launch Cloud Node →Pre-configured environment

Frequently Asked Questions (FAQ)

Model weights are only one part of total VRAM consumption. When you prompt a model, the KV Cache (Key-Value Cache) stores all historical tokens. At 32k or 64k context, standard GQA attention can demand an additional 4GB to 12GB of VRAM! Combined with activation buffers and OS background consumption (0.8GB on Windows), total memory spikes beyond hardware capacity.

GPU Model	Architecture	VRAM Capacity	Memory Bandwidth	TDP	Max Runnable Model Tier	Interactive Simulator
NVIDIA GeForce RTX 5090	Blackwell	32 GB	1792 GB/s	600W	70B Native / 671B Quantized	Benchmark
NVIDIA GeForce RTX 5080	Blackwell	16 GB	1024 GB/s	400W	Qwen 2.5 Coder 32B / 14B Native	Benchmark
NVIDIA GeForce RTX 5070 Ti	Blackwell	16 GB	896 GB/s	300W	Qwen 2.5 Coder 32B / 14B Native	Benchmark
NVIDIA GeForce RTX 5070	Blackwell	12 GB	672 GB/s	250W	DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)	Benchmark
NVIDIA GeForce RTX 4090	Ada Lovelace	24 GB	1008 GB/s	450W	DeepSeek-R1 32B Sweet Spot (Full Speed)	Benchmark
NVIDIA GeForce RTX 4080 Super	Ada Lovelace	16 GB	736 GB/s	320W	Qwen 2.5 Coder 32B / 14B Native	Benchmark
NVIDIA GeForce RTX 4080	Ada Lovelace	16 GB	717 GB/s	320W	Qwen 2.5 Coder 32B / 14B Native	Benchmark
NVIDIA GeForce RTX 4070 Ti Super	Ada Lovelace	16 GB	672 GB/s	285W	Qwen 2.5 Coder 32B / 14B Native	Benchmark
NVIDIA GeForce RTX 4070 Ti	Ada Lovelace	12 GB	504 GB/s	285W	DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)	Benchmark
NVIDIA GeForce RTX 4070 Super	Ada Lovelace	12 GB	504 GB/s	220W	DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)	Benchmark

GPU Model

Architecture

VRAM Capacity

Memory Bandwidth

TDP

Max Runnable Model Tier

Interactive Simulator

NVIDIA GeForce RTX 5090

Blackwell

32 GB

1792 GB/s

600W

70B Native / 671B Quantized

Benchmark

NVIDIA GeForce RTX 5080

Blackwell

16 GB

1024 GB/s

400W

Qwen 2.5 Coder 32B / 14B Native

Benchmark

NVIDIA GeForce RTX 5070 Ti

Blackwell

16 GB

896 GB/s

300W

Qwen 2.5 Coder 32B / 14B Native

Benchmark

NVIDIA GeForce RTX 5070

Blackwell

12 GB

672 GB/s

250W

DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)

Benchmark

NVIDIA GeForce RTX 4090

Ada Lovelace

24 GB

1008 GB/s

450W

DeepSeek-R1 32B Sweet Spot (Full Speed)

Benchmark

NVIDIA GeForce RTX 4080 Super

Ada Lovelace

16 GB

736 GB/s

320W

Qwen 2.5 Coder 32B / 14B Native

Benchmark

NVIDIA GeForce RTX 4080

Ada Lovelace

16 GB

717 GB/s

320W

Qwen 2.5 Coder 32B / 14B Native

Benchmark

NVIDIA GeForce RTX 4070 Ti Super

Ada Lovelace

16 GB

672 GB/s

285W

Qwen 2.5 Coder 32B / 14B Native

Benchmark

NVIDIA GeForce RTX 4070 Ti

Ada Lovelace

12 GB

504 GB/s

285W

DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)

Benchmark

NVIDIA GeForce RTX 4070 Super

Ada Lovelace

12 GB

504 GB/s

220W

DeepSeek-R1 14B & FLUX.1 [dev] (12GB King)

Benchmark

Model Name	Architecture	Parameters	Native Context	Q4_K_M VRAM	FP16 VRAM	Min Recommended GPU
DeepSeek-R1 (671B Full MoE)	DeepSeek MLA	671B	128k Tokens	~467.0 GB	~1610.4 GB	Mac Studio M2 Ultra (192GB)
DeepSeek-V3 (671B Base/Chat)	DeepSeek MLA	671B	128k Tokens	~467.0 GB	~1610.4 GB	Mac Studio M2 Ultra (192GB)
DeepSeek-R1-Distill-Llama-70B	GQA	70.6B	128k Tokens	~49.1 GB	~169.4 GB	2x RTX 3090 / RTX 5090 32G
DeepSeek-R1-Distill-Qwen-32B	GQA	32.8B	128k Tokens	~22.8 GB	~78.7 GB	RTX 4090 24G / RTX 3090 24G
DeepSeek-R1-Distill-Qwen-14B	GQA	14.7B	128k Tokens	~10.2 GB	~35.3 GB	RTX 3060 12G / 4070 12G
DeepSeek-R1-Distill-Llama-8B	GQA	8B	128k Tokens	~5.6 GB	~19.2 GB	RTX 3060 12G / 4060 8G
DeepSeek-R1-Distill-Qwen-8B	GQA	8.03B	128k Tokens	~5.6 GB	~19.3 GB	RTX 3060 12G / 4070 12G
DeepSeek-R1-Distill-Qwen-7B	GQA	7.6B	128k Tokens	~5.3 GB	~18.2 GB	RTX 3060 12G / 4060 8G

Model Name

Architecture

Parameters

Native Context

Q4_K_M VRAM

FP16 VRAM

Min Recommended GPU

DeepSeek-R1 (671B Full MoE)

DeepSeek MLA

671B

128k Tokens

~467.0 GB

~1610.4 GB

Mac Studio M2 Ultra (192GB)

DeepSeek-V3 (671B Base/Chat)

DeepSeek MLA

671B

128k Tokens

~467.0 GB

~1610.4 GB

Mac Studio M2 Ultra (192GB)

DeepSeek-R1-Distill-Llama-70B

GQA

70.6B

128k Tokens

~49.1 GB

~169.4 GB

2x RTX 3090 / RTX 5090 32G

DeepSeek-R1-Distill-Qwen-32B

GQA

32.8B

128k Tokens

~22.8 GB

~78.7 GB

RTX 4090 24G / RTX 3090 24G

DeepSeek-R1-Distill-Qwen-14B

GQA

14.7B

128k Tokens

~10.2 GB

~35.3 GB

RTX 3060 12G / 4070 12G

DeepSeek-R1-Distill-Llama-8B

GQA

128k Tokens

~5.6 GB

~19.2 GB

RTX 3060 12G / 4060 8G

DeepSeek-R1-Distill-Qwen-8B

GQA

8.03B

128k Tokens

~5.6 GB

~19.3 GB

RTX 3060 12G / 4070 12G

DeepSeek-R1-Distill-Qwen-7B

GQA

7.6B

128k Tokens

~5.3 GB

~18.2 GB

RTX 3060 12G / 4060 8G

Can Your Hardware Run This AI Model?

Can Run Perfectly

Hardware Diagnostics & Benchmark Suite

Hardware Compatibility Matrix

Dual-GPU Showdown Arena

Best GPU Finder (Reverse Matcher)

Fine-Tune or Scale DeepSeek-R1-Distill-Qwen-14B on NVIDIA RTX 4090 (24GB)

Frequently Asked Questions (FAQ)

Authoritative AI Hardware Reference Benchmark (Top GPUs)

Open-Weights LLM & Diffusion Model VRAM Reference Directory

Mathematical Methodology: GQA vs DeepSeek MLA

Curated AI Hardware & Model Compatibility Matrix Directory