BENCHMARK REPORT
Can I Run DeepSeek-R1-Distill-Qwen-8B on Intel Gaudi 3 (128GB)?
Comprehensive technical verdict based on static weights, dynamic KV Cache (8k baseline context window), and CUDA runtime overhead.
Official Verdict
YES — Runs Smoothly
Executive Hardware Compatibility Summary: DeepSeek-R1-Distill-Qwen-8B on Intel Gaudi 3 (128GB)
Authoritative verdict, memory allocation, and token generation performance:
Direct Answer: Can Run PerfectlyDeepSeek-R1-Distill-Qwen-8B runs smoothly with 100% dedicated VRAM allocation on Intel Gaudi 3 (128GB) (128GB). At standard Q4_K_M 4-bit quantization and an 8,192-token context window, total memory demand is 7.48 GB (4.89 GB weights + 1.19 GB dynamic KV cache + 1.15 GB OS & CUDA buffers). Physical GPU memory provides 120.52 GB of free safety buffer, preventing PCIe memory bus throttling. Estimated generation throughput is ~180 tokens/sec (~5.6 ms per token) with prompt prefill processing reaching ~4800 tok/s.
Run Verdict
100% Fit (Optimal)
Total Required VRAM
7.48 GB / 128 GB
Decoding Speed
~180 tok/s (5.6ms)
Safe Context Window
Up to 32k tokens
Hardware Architecture & VRAM Topology
GaudiIntel Gaudi 3 (128GB) • 3700 GB/s Bandwidth
Token Generation
~180 tok/s(5.6 ms)
OPTIMAL
VRAM Utilization Breakdown7.48 GB / 128 GB (6%)
+120.52G Free
Weights
4.89 GB
KV Cache
1.19 GB
Activation
0.25 GB
OS/Driver
1.15 GB
Free Headroom
120.52 GB
Max Safe Context Limit on this Hardware
You can safely expand prompts up to 131,072 tokens before CUDA OOM.
Headroom Used
8,192 / 131,072
Context Window vs VRAM Scaling Curve
Blue line indicates total VRAM demand across context lengths; Red dashed line is physical VRAM ceiling.Max Safe: 131,072 tokens
Can Run Perfectly
optimalEntire model & KV cache fit fully into VRAM with pure GPU acceleration.
Throughput
~180 tok/s
5.6 ms / token
Headroom
120.52 GB
6% of capacity
Layers Fit
36 / 36
100% in VRAM
Max Safe Ctx
128k
Tokens before OOM
Static Permalink:
Open Static Report/can-i-run/deepseek-r1-distill-qwen-8b-on-gaudi-3Ready-to-Run CLI Commands
ollama run deepseek-r1:8bCommands include automatic memory ceiling flags and model layer offload configurations calculated above.
Cluster Scaled Serving
gpuy.dev On-Demand ComputeFine-Tune or Scale DeepSeek-R1-Distill-Qwen-8B on NVIDIA RTX 4090 (24GB)
Local inference runs well, but LoRA fine-tuning and multi-user vLLM serving require 40GB+ VRAM with FP8/FP16 optimizer states. Spin up dedicated clusters starting from $0.35/hr.
2x Faster Speed
$0.35/hr
Breakeven: ~5,286 hrs vs hardware purchase
Per-second billing
Launch Cloud Node →Pre-configured environment
Hardware Specification Summary
GPU ArchitectureGaudi
Memory Bandwidth3700 GB/s
Model Parameters8.03B (LLM)
Attention ArchitectureGrouped-Query (GQA)
Prompt Prefill Speed~4800 tok/s (TTFT ~1707ms)
Benchmark Context Tested8,192 tokens
Global Compatibility Matrix for Intel Gaudi 3 (128GB)
Evaluated at Standard Q4_K_M Precision with 8,192 Tokens Context Window
Compatibility Index
93% (53/57)
53 Pass0 Tight4 OOM
| Model & Architecture | Category | Parameters | Required VRAM | Fit Status | Est. Speed | Simulate |
|---|---|---|---|---|---|---|
DeepSeek-R1 (671B Full MoE)MLAMoE DeepSeek | MoE | 671B(37B active) | 411.16 GB | OOM (-283.16G) | ~15.7 tok/s | Simulate |
DeepSeek-V3 (671B Base/Chat)MLAMoE DeepSeek | MoE | 671B(37B active) | 411.16 GB | OOM (-283.16G) | ~15.7 tok/s | Simulate |
DeepSeek-R1-Distill-Llama-70B DeepSeek | LLM | 70.6B | 47.6 GB | 100% VRAM Fit | ~60.2 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-32B DeepSeek | LLM | 32.8B | 23.75 GB | 100% VRAM Fit | ~129.7 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-14B DeepSeek | LLM | 14.7B | 11.94 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-R1-Distill-Llama-8B DeepSeek | LLM | 8B | 7.33 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-8B DeepSeek | LLM | 8.03B | 7.48 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-7B DeepSeek | LLM | 7.6B | 6.49 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-1.5B DeepSeek | LLM | 1.8B | 2.61 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-Coder-V2 Lite (16B / 2.4B MoE)MLAMoE DeepSeek | MoE | 15.7B(2.4B active) | 11.21 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-Coder-V2 (236B / 21B MoE)MLAMoE DeepSeek | MoE | 236B(21B active) | 146.23 GB | OOM (-18.23G) | ~80.3 tok/s | Simulate |
Qwen 2.5 Coder 72B Instruct Qwen | LLM | 72.7B | 48.87 GB | 100% VRAM Fit | ~58.5 tok/s | Simulate |
Qwen 2.5 72B Instruct Qwen | LLM | 72.7B | 48.87 GB | 100% VRAM Fit | ~58.5 tok/s | Simulate |
Qwen 2.5 VL 72B (Vision Flagship) Qwen | LLM | 72.7B | 48.87 GB | 100% VRAM Fit | ~58.5 tok/s | Simulate |
Qwen 2.5 VL 7B (Vision) Qwen | LLM | 7.61B | 6.5 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 Coder 32B Instruct Qwen | LLM | 32.8B | 23.75 GB | 100% VRAM Fit | ~129.7 tok/s | Simulate |
Qwen 2.5 Coder 14B Instruct Qwen | LLM | 14.7B | 11.94 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 Coder 7B Instruct Qwen | LLM | 7.6B | 6.49 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 Coder 1.5B Instruct Qwen | LLM | 1.54B | 2.45 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 32B Instruct Qwen | LLM | 32.8B | 23.75 GB | 100% VRAM Fit | ~129.7 tok/s | Simulate |
Qwen 2.5 14B Instruct Qwen | LLM | 14.7B | 11.94 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 7B Instruct Qwen | LLM | 7.6B | 6.49 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 3B Instruct Qwen | LLM | 3.1B | 3.59 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Qwen 2.5 0.5B Instruct Qwen | LLM | 0.49B | 1.8 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Llama 3.3 70B Instruct Llama | LLM | 70.6B | 47.6 GB | 100% VRAM Fit | ~60.2 tok/s | Simulate |
Llama 3.1 70B Instruct Llama | LLM | 70.6B | 47.6 GB | 100% VRAM Fit | ~60.2 tok/s | Simulate |
Llama 3.1 405B Instruct Llama | LLM | 405B | 252.77 GB | OOM (-124.77G) | ~2.4 tok/s | Simulate |
Llama 3.1 8B Instruct Llama | LLM | 8B | 7.33 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Llama 3.2 3B Instruct Llama | LLM | 3.2B | 4.28 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Llama 3.2 1B Instruct Llama | LLM | 1.23B | 2.41 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Llama 3.2 11B Vision Instruct Llama | LLM | 10.6B | 8.92 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Llama 3.2 90B Vision Instruct Llama | LLM | 89B | 59.46 GB | 100% VRAM Fit | ~47.8 tok/s | Simulate |
Microsoft Phi-4 (14B) Phi | LLM | 14.7B | 12.01 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Microsoft Phi-3.5 Mini (3.8B) Phi | LLM | 3.82B | 6.91 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Google Gemma 2 9B Instruct Gemma | LLM | 9.2B | 9.79 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Mistral Small 3 (24B Instruct) Mistral | LLM | 24B | 18.12 GB | 100% VRAM Fit | ~177.2 tok/s | Simulate |
Mistral NeMo (12B Instruct) Mistral | LLM | 12.2B | 10.15 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Mistral 7B Instruct v0.3 Mistral | LLM | 7.25B | 6.88 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Codestral 22B (Mistral AI) Mistral | LLM | 22.2B | 17.02 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Mistral Large 2 (123B) Mistral | LLM | 123B | 79.77 GB | 100% VRAM Fit | ~34.6 tok/s | Simulate |
Mixtral 8x7B (MoE 47B)MoE Mistral | MoE | 46.7B(12.9B active) | 31.15 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Mixtral 8x22B (MoE 141B)MoE Mistral | MoE | 141B(39B active) | 89.67 GB | 100% VRAM Fit | ~109 tok/s | Simulate |
Pixtral 12B (Multimodal) Mistral | LLM | 12.4B | 10.28 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Gemma 2 27B Instruct Gemma | LLM | 27.2B | 21.26 GB | 100% VRAM Fit | ~156.4 tok/s | Simulate |
Google Gemma 2 2B Instruct Gemma | LLM | 2.6B | 3.84 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
GLM-4 9B Chat (Zhipu AI) GLM | LLM | 9.4B | 7.79 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
FLUX.1 [dev] (12B DiT) FLUX | ImageGen | 12B | 15.68 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
FLUX.1 [schnell] (4-step) FLUX | ImageGen | 12B | 15.68 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Stable Diffusion XL (SDXL 1.0) SDXL | ImageGen | 3.5B | 7.51 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Stable Diffusion 1.5 (Classic) SD | ImageGen | 1B | 5.78 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Whisper Large v3 (1.5B) Whisper | Audio | 1.5B | 3.64 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Kokoro 82M (Text-to-Speech) Kokoro | Audio | 0.082B | 1.56 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Wan2.1 (14B Text-to-Video) Wan | ImageGen | 14.2B | 18.72 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Wan2.1 (1.3B Video) Wan | ImageGen | 1.3B | 7.28 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
HunyuanVideo (13B Video DiT) Hunyuan | ImageGen | 13B | 19.69 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
Stable Diffusion 3.5 Large (8B) SD | ImageGen | 8.1B | 13.3 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
BAAI BGE-M3 (Multilingual Embedding) BAAI | LLM | 0.56B | 2.54 GB | 100% VRAM Fit | ~180 tok/s | Simulate |
DeepSeek-R1-Distill-Qwen-8B Context Scaling Matrix on Intel Gaudi 3 (128GB)
Dynamic KV Cache footprint and generation throughput measured across sequence lengths (Q4_K_M weights):
| Context Window | KV Cache Memory | Total VRAM Demand | Compatibility Status | Offloaded Layers | Est. Generation Speed |
|---|---|---|---|---|---|
| 2,048 tokens (2k) | 0.30 GB | 6.59 GB / 128 GB | 100% Fit | 0 (All in VRAM) | ~180.0 tok/s |
| 4,096 tokens (4k) | 0.60 GB | 6.89 GB / 128 GB | 100% Fit | 0 (All in VRAM) | ~180.0 tok/s |
| 8,192 tokens (8k)Standard Baseline | 1.19 GB | 7.48 GB / 128 GB | 100% Fit | 0 (All in VRAM) | ~180.0 tok/s |
| 16,384 tokens (16k) | 2.39 GB | 8.68 GB / 128 GB | 100% Fit | 0 (All in VRAM) | ~180.0 tok/s |
| 32,768 tokens (32k) | 4.77 GB | 11.06 GB / 128 GB | 100% Fit | 0 (All in VRAM) | ~180.0 tok/s |
Hardware Compatibility Q&A: DeepSeek-R1-Distill-Qwen-8B on Intel Gaudi 3 (128GB)
Technical answers for memory allocation, CUDA overhead, and execution parameters.
Can Intel Gaudi 3 (128GB) run DeepSeek-R1-Distill-Qwen-8B?
Can Run Perfectly. Entire model & KV cache fit fully into VRAM with pure GPU acceleration. At an 8k context window, total VRAM demand is 7.48 GB compared to Intel Gaudi 3 (128GB)'s 128 GB frame buffer.
How much VRAM does DeepSeek-R1-Distill-Qwen-8B require?
At standard Q4_K_M quantization and 8k context, DeepSeek-R1-Distill-Qwen-8B requires approximately 7.48 GB of VRAM. This includes 4.89 GB for model weights, 1.19 GB for dynamic KV cache (Grouped-Query Attention (GQA)), 0.25 GB for activation buffers, and 1.15 GB for OS display and CUDA runtime overhead.
What generation speed (tokens/sec) can I expect on Intel Gaudi 3 (128GB)?
Estimated autoregressive token generation speed is ~180 tokens per second with a per-token latency of ~5.6 ms, powered by 3700 GB/s memory bandwidth. Prompt prefill processing reaches ~4800 tok/s (TTFT ~1707 ms).
What causes CUDA Out-of-Memory (OOM) when running DeepSeek-R1-Distill-Qwen-8B?
CUDA OOM occurs when total memory consumption exceeds Intel Gaudi 3 (128GB)'s 128 GB physical VRAM. While model weights stay static, the KV cache grows with every conversation token. When VRAM fills up, inference engines (like Ollama or llama.cpp) must offload remaining layers across the PCIe bus (31.5 GB/s) to host system RAM, causing generation speed to drop sharply.
How do I run DeepSeek-R1-Distill-Qwen-8B locally on Intel Gaudi 3 (128GB)?
You can run DeepSeek-R1-Distill-Qwen-8B directly with Ollama using "ollama run deepseek-r1-distill-qwen-8b". For advanced serving with vLLM or llama.cpp, copy the ready-to-run CLI commands in the CLI Commands section above with optimal context, quantization, and GPU layer offload parameters.
Related Hardware Benchmarks & Model Alternatives
Compare DeepSeek-R1-Distill-Qwen-8B and Intel Gaudi 3 (128GB) against alternative configurations:
Recommended Alternative GPUs for DeepSeek-R1-Distill-Qwen-8B
Explore how DeepSeek-R1-Distill-Qwen-8B performs across other budget, midrange, and flagship architectures:
Other Popular Models You Can Run on Intel Gaudi 3 (128GB)
Verified open-weights LLMs and image models that fit within Intel Gaudi 3 (128GB)'s 128GB VRAM:
Dual GPU Showdown: Intel Gaudi 3 (128GB) vs NVIDIA GeForce RTX 4090
Compare memory bandwidth (3700 GB/s vs 1008 GB/s), VRAM headroom (128GB vs 24GB), and inference speed head-to-head.
Frequently Asked Questions (FAQ)
Model weights are only one part of total VRAM consumption. When you prompt a model, the KV Cache (Key-Value Cache) stores all historical tokens. At 32k or 64k context, standard GQA attention can demand an additional 4GB to 12GB of VRAM! Combined with activation buffers and OS background consumption (0.8GB on Windows), total memory spikes beyond hardware capacity.