LLM GPU VRAM Requirements Explained
Trying to run a local LLM only to hit an "out of memory" error is frustrating. I've been there - staring at 8GB of VRAM that refuses to load a 13B model. Understanding VRAM requirements is essential before investing in hardware or downloading models.
The quick answer: VRAM stores LLM model weights during inference. A 7B parameter model needs approximately 4GB VRAM at 4-bit quantization, while a 70B model requires around 40GB. The formula is simple - multiply parameters by bits per parameter, divide by 8, then add overhead for the KV cache and activations.
After testing dozens of models across different GPUs over the past two years, I've learned that VRAM calculations aren't always straightforward. Quantization, context windows, and optimization techniques all play major roles. This guide breaks down exactly what you need to know.
Key Takeaway: "Most users can run 7B models on 8GB VRAM cards using 4-bit quantization. For 13B models, you need at least 10-12GB VRAM. 70B models require 40GB+ or multi-GPU setups."
In this guide, you'll learn the exact formulas for calculating VRAM needs, how quantization affects memory requirements, and practical techniques for running larger models on limited hardware. Check out our guide to the best GPUs for local LLMs once you understand your requirements.
Quick Reference: VRAM by Model Size
Before diving into the details, here's a quick reference table showing approximate VRAM requirements for common model sizes at different quantization levels.
| Model Size | 4-bit VRAM | 8-bit VRAM | FP16 VRAM | Typical Use Case |
|---|---|---|---|---|
| 1B - 3B | 1-2 GB | 2-3 GB | 4-6 GB | Basic tasks, testing |
| 7B | 4-5 GB | 7-8 GB | 14-16 GB | General chat, coding |
| 13B | 7-8 GB | 13-14 GB | 26-28 GB | Better reasoning |
| 34B | 18-20 GB | 34-36 GB | 68-72 GB | Advanced tasks |
| 70B | 38-42 GB | 70-75 GB | 140+ GB | Near-GPT performance |
These estimates include model weights plus approximately 10-20% overhead for the KV cache and activation memory. Your actual requirements may vary based on context length and the software you use.
What is VRAM and Why Does It Matter for LLMs?
VRAM (Video RAM) is dedicated memory on your GPU that stores model weights and activations during LLM inference. Without sufficient VRAM, you cannot load or run large language models locally.
Unlike system RAM, VRAM is physically located on the graphics card and connected directly to the GPU. This proximity enables extremely fast data transfer rates - essential for LLM inference where the model reads parameters billions of times per second.
When you load an LLM, the entire model must fit in VRAM at once. There's no swapping to system RAM like there is with regular applications. If your GPU has 8GB VRAM and the model requires 9GB, you simply cannot run it without special techniques.
Model Parameters: The weights in a neural network that determine its behavior. A 7B model has 7 billion parameters. Each parameter must be stored in memory during inference.
I learned this the hard way when I tried loading a 30B model on my RTX 3080 Ti. Despite having 12GB VRAM, the unquantized model needed over 60GB. This led me to discover quantization, which I'll explain in detail later.
The VRAM Calculation Formula for LLMs
The basic formula for LLM VRAM requirements is: VRAM (GB) = (Parameters x Bits per parameter) + 8 + Overhead. For a 7B model at 4-bit: (7,000,000,000 x 4) + 8 = approximately 3.5GB for weights, plus 1-2GB overhead.
Let me break this down step by step. Understanding this formula will help you calculate requirements for any model.
Step 1: Calculate Parameter Storage
- Start with parameter count: A 7B model has 7,000,000,000 parameters
- Multiply by bits per parameter: 4-bit = 4 bits, FP16 = 16 bits
- Convert to bytes: Divide by 8 (8 bits = 1 byte)
- Convert to gigabytes: Divide by 1,073,741,824 (or roughly 1 billion)
Step 2: Add Overhead
The KV cache and activation memory add overhead. For shorter contexts (2K-4K tokens), budget 10-20% extra. For long contexts (8K+), budget 30-50% extra.
Real Calculation Example
Llama 2 7B at 4-bit quantization:
- Parameters: 7,000,000,000
- Bits per parameter: 4
- Bytes: 7,000,000,000 x 4 / 8 = 3,500,000,000 bytes
- Gigabytes: 3,500,000,000 / 1,073,741,824 = ~3.26 GB
- Overhead (15%): ~0.5 GB
- Total: ~3.75-4 GB VRAM
This explains why 7B models run comfortably on 8GB cards. I've run this exact setup on an RTX 3060 with room to spare for context and other applications.
Understanding Quantization and VRAM Impact
Quantization reduces model precision from FP16 (16-bit) to lower bit formats like 8-bit or 4-bit. This reduces VRAM requirements by 50-75% with minimal quality loss, making large models accessible on consumer hardware.
Quantization is the single most important technique for running LLMs on limited VRAM. It reduces the precision of model weights, requiring fewer bits per parameter.
| Format | Bits/Param | VRAM for 7B | VRAM for 70B | Quality Impact |
|---|---|---|---|---|
| FP32 | 32 | 28 GB | 280 GB | Baseline (training quality) |
| FP16 | 16 | 14 GB | 140 GB | No perceptible loss |
| 8-bit (INT8) | 8 | 7 GB | 70 GB | Minimal loss for most tasks |
| 4-bit (Q4_K_M) | 4 | 4 GB | 35 GB | Slight loss, excellent value |
| 4-bit (EXL2) | 4.5 | 4.5 GB | 40 GB | Optimized for speed |
GGUF Format: A file format designed for llama.cpp that stores quantized models efficiently. GGUF files are the standard for running LLMs on consumer hardware and support various quantization levels.
I've tested all these quantization levels extensively. For casual use, 4-bit models offer the best balance. The difference between Q4_K_M and FP16 is barely noticeable in chat responses, but the VRAM savings are enormous.
Use 4-bit (Q4_K_M) For
General chat, coding assistance, and most use cases. Best VRAM efficiency with minimal quality loss.
Use FP16/8-bit For
Critical tasks, benchmarking, or when you have abundant VRAM. Maximum quality at the cost of memory.
Popular LLM VRAM Requirements
Different model families have different VRAM requirements even at the same parameter count. Here are the requirements for popular models as of 2026.
Llama Family (Meta)
| Model | 4-bit VRAM | 8-bit VRAM | Notes |
|---|---|---|---|
| Llama 3.2 3B | 2 GB | 3 GB | Runs on any GPU, great for testing |
| Llama 3.1 8B | 5 GB | 8 GB | Perfect for 8GB cards at 4-bit |
| Llama 2/3 13B | 8 GB | 14 GB | Needs 12GB+ for comfortable 4-bit use |
| Llama 3.1 70B | 40 GB | 75 GB | Requires dual 3090s or A6000 |
| Llama 3.1 405B | 230 GB | 430 GB | Multi-GPU cluster required |
Mistral and Mixtral (Mistral AI)
| Model | 4-bit VRAM | 8-bit VRAM | Notes |
|---|---|---|---|
| Mistral 7B | 5 GB | 8 GB | Excellent performance, runs on 8GB cards |
| Mixtral 8x7B | 26 GB | 47 GB | Mixture of Experts, behaves like 47B |
| Mixtral 8x22B | 82 GB | 141 GB | Enterprise hardware required |
I've personally run Mistral 7B on an RTX 3060 12GB, and it performs exceptionally well at 4-bit quantization. The model punches above its weight class compared to Llama 2 7B.
If you're planning a GPU purchase around these requirements, check out our high VRAM GPU models guide or consider budget GPU options if you're working with limited funds. AMD users should review our AMD card recommendations.
How Context Windows and KV Cache Affect VRAM?
Context windows significantly impact VRAM. The KV cache grows with context length - approximately 0.5-1GB extra VRAM per 1000 tokens for 7B models. A 4K context needs ~2-4GB more VRAM than the base model size.
The KV cache stores attention keys and values for each token in your conversation. As your context grows, so does the memory requirement.
KV Cache: Key-Value cache that stores computed attention states during generation. This cache enables faster responses but increases with context length. A 4K context requires significantly more VRAM than a 512-token context.
Here's how context length affects VRAM for a 7B model at 4-bit:
- 512 tokens: ~0.3 GB KV cache
- 2K tokens: ~1.2 GB KV cache
- 4K tokens: ~2.5 GB KV cache
- 8K tokens: ~5 GB KV cache
- 16K tokens: ~10 GB KV cache
- 32K tokens: ~20 GB KV cache
This is why a model that loads fine might crash when you extend the context window. I experienced this repeatedly when testing long-context models on my 12GB card - the model would load at 4K context but fail at 16K.
Pro Tip: When monitoring VRAM usage during long conversations, the KV cache will gradually increase. Use our guide on checking per-process VRAM usage to track memory consumption in real-time.
Running Larger Models on Limited VRAM
Don't have enough VRAM for your desired model? Several techniques can help you run larger models on smaller GPUs.
CPU Offloading
CPU offloading moves some model layers to system RAM, reducing GPU VRAM requirements. The tradeoff is significantly slower inference - I've seen 5-10x speed reductions when offloading more than 20% of a model.
Typical VRAM savings with CPU offloading:
- 20% offload: saves ~20% VRAM, ~2x slower
- 50% offload: saves ~50% VRAM, ~5x slower
- 80% offload: saves ~80% VRAM, ~10-20x slower
This technique lets me run 13B models on my 8GB GPU, but the 3-5 token per second speed is painful for anything beyond testing.
GGUF and llama.cpp
The llama.cpp project with GGUF format models offers the best VRAM efficiency. These models are specifically quantized for consumer hardware and include advanced offloading options.
Multi-GPU Setups
For 70B+ models, multi-GPU setups are often necessary. Tensor parallelism splits the model across multiple GPUs. Two RTX 3090s (24GB each) can run 70B models comfortably at 4-bit.
See our guide on multi-GPU setups for detailed information on splitting models across multiple GPUs.
Flash Attention and Other Optimizations
Modern attention mechanisms like Flash Attention reduce KV cache memory requirements. These optimizations can save 20-30% VRAM for long contexts without quality loss. Our guide on freeing up GPU memory covers additional optimization techniques.
| Your VRAM | Max Model (4-bit) | Recommended Models |
|---|---|---|
| 6 GB | 3B | Llama 3.2 3B, Phi-3 Mini |
| 8 GB | 7-8B | Llama 3.1 8B, Mistral 7B |
| 12 GB | 13B | Llama 2/3 13B, Mixtral 8x7B (heavy offload) |
| 16 GB | 20B | Command R, Yi 34B (offloaded) |
| 24 GB | 34B | Yi 34B, Mixtral 8x7B |
| 48 GB | 70B | Llama 3.1 70B |
For software recommendations compatible with your hardware, check out our list of local LLM software and learn about CUDA and alternatives.
Frequently Asked Questions
How much VRAM do I need for LLM?
For basic LLM use, 8GB VRAM runs 7B models comfortably at 4-bit quantization. For 13B models, you need 10-12GB VRAM. Large 70B models require 40GB+ VRAM or multi-GPU setups. Most users start with 8-12GB cards.
Can I run a 13B model on 8GB VRAM?
Yes, with 4-bit quantization and CPU offloading enabled. You'll get approximately 7-8GB used for the model itself, leaving room for context. However, expect slower performance (3-5 tokens/second) compared to running on a 12GB+ card.
Does context length affect VRAM?
Yes, context length significantly affects VRAM through the KV cache. For a 7B model, each 1000 tokens adds approximately 0.5-1GB of VRAM usage. A 4K context might need 2-4GB more VRAM than the base model size.
What is quantization in LLM?
Quantization reduces the precision of model weights, typically from 16-bit (FP16) to 4-bit or 8-bit formats. This reduces VRAM requirements by 50-75% with minimal quality loss. 4-bit quantization is the standard for running large models on consumer GPUs.
How much VRAM for Llama 3 70B?
Llama 3 70B requires approximately 40GB VRAM at 4-bit quantization. At 8-bit, it needs about 75GB. This means you need either an enterprise GPU (RTX A6000, A100) or a multi-GPU setup with two 24GB cards to run it.
Can I run LLM on CPU instead of GPU?
Yes, but performance is significantly slower. CPU inference runs at 1-5 tokens per second versus 50+ tokens per second on GPU. CPU offloading lets you run larger models on smaller GPUs by moving some layers to system RAM, but this also reduces speed.
What's the difference between FP16 and 4-bit quantization?
FP16 uses 16 bits per parameter and is the standard precision. 4-bit uses only 4 bits per parameter, reducing VRAM requirements by 75%. The quality difference is minimal for most use cases, making 4-bit the preferred choice for consumer hardware.
What is GGUF format?
GGUF is a file format designed for llama.cpp that stores quantized LLMs efficiently. It supports various quantization levels (Q2_K through Q6_K) and includes metadata for CPU/GPU offloading. GGUF is the standard format for running LLMs on consumer hardware.
Final Recommendations
After spending two years running LLMs on various hardware configurations, here's what I recommend based on your situation:
VRAM Recommendations by Use Case
8GB VRAM
Runs 7B models comfortably, great for learning
12-16GB VRAM
Runs 13B models, allows longer contexts
24GB+ VRAM
Runs 34B models, 70B with multi-GPU
The key insight I've learned is that model quality matters more than size. A well-quantized 7B model often outperforms a poorly quantized 13B model. Start with what your hardware can handle comfortably, and upgrade only when you hit genuine limitations.
Remember that VRAM isn't the only factor. System RAM, CPU speed, and storage (SSD vs HDD) all affect the overall experience. But VRAM remains the hard limit for which models you can load.