Local LLM VRAM, KV Cache & Decode Bandwidth Calculator
Calculate exact Transformer weight footprint (GiB), Grouped-Query Attention (GQA) KV cache memory across parallel slots, runtime workspace overhead, single-stream Decode tok/s, and copy-ready CLI flags for Ollama, vLLM, and llama.cpp.
1. Model, Quantization & Runtime Controls
2. Real-Time VRAM Budget, Decode tok/s & 6-Target Fit Matrix
Simultaneous 6-Hardware Target Comparison Matrix
Click any row to select target| Hardware Target | Usable Ceiling | Utilization | Decode tok/s (1-Stream) | Fit Verdict |
|---|---|---|---|---|
| RTX 5070 Ti 16GB 896 GB/s GDDR7 | 16.0 GiB | 149.1% | ~30.1 tok/s (max 45.1) | OOM: Exceeds Capacity |
| RTX 3090 / 4090 24GB 1008 GB/s GDDR6X | 24.0 GiB | 99.4% | ~33.9 tok/s (max 50.8) | Tight Fit (OS/Display Headroom Warning) |
| Mac mini M4/M4 Pro 32GB 273 GB/s · 28 GiB wired max | 24.0 GiB | 99.4% | ~9.2 tok/s (max 13.7) | Tight Fit (OS/Display Headroom Warning) |
| Mac mini M4 Pro 64GB 273 GB/s · 56 GiB wired max | 48.0 GiB | 49.7% | ~9.2 tok/s (max 13.7) | 100% GPU Fit |
| NVIDIA GB10 128GB UMA GX10 / AI TOP ATOM / DGX Spark · 273 GB/s | 118.0 GiB | 20.2% | ~9.2 tok/s (max 13.7) | 100% GPU Fit |
| Dual GPU 48GB (2x 24GB TP=2) 1750 GB/s effective TP=2 | 48.0 GiB | 49.7% | ~58.8 tok/s (max 88.1) | 100% GPU Fit |
Production Launch CLI (Ollama systemd, vLLM & llama-server)
# Target: Qwen 2.5 / Qwen 3 32B (Q4_K_M, 32768 ctx, KV=q8_0, Slots=1) -> 23.86 GiB
# 1. Ollama CLI & systemd drop-in (/etc/systemd/system/ollama.service.d/override.conf)
# [Service]
# Environment="OLLAMA_FLASH_ATTENTION=1" "OLLAMA_KV_CACHE_TYPE=q8_0" "OLLAMA_NUM_PARALLEL=1" "OLLAMA_CONTEXT_LENGTH=32768"
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_NUM_PARALLEL=1 ollama run qwen2.5:32b
# 2. vLLM OpenAI-Compatible Server (PagedAttention + CUDA Graph)
vllm serve Qwen/Qwen2.5-32B-Instruct --max-model-len 32768 --max-num-seqs 1 --kv-cache-dtype fp8 --gpu-memory-utilization 0.90
# 3. llama.cpp Native HTTP Server (GGUF + Flash Attention)
llama-server -m ./models/Qwen2.5-32B-Instruct-Q4_K_M.gguf -ngl 99 -fa -ctk q8_0 -ctv q8_0 -c 32768 -np 1 --host 127.0.0.1 --port 8080
Exact Transformer VRAM & GQA KV Cache Equations
Autoregressive LLM memory splits into three distinct allocations: static quantized weights, dynamic Key-Value (KV) attention state, and runtime compute buffers. Unlike legacy Multi-Head Attention, modern architectures such as Llama 3.3 70B and Qwen 32B use Grouped-Query Attention (GQA) with 8 KV heads instead of 64 query heads, shrinking per-token cache footprint by 8x. For each active token across slots concurrent requests, KV memory equals 2 * layers * kv_heads * head_dim * bytes_per_element.
In llama.cpp K-quant formats, super-block scales add fractional overhead above nominal bit width (for example, Q4_K_M averages ~4.85 bpw and Q8_0 uses 34 bytes per 32-element block, or 8.50 bpw / 1.0625 B/elem). Passing -fa -ctk q8_0 -ctv q8_0 to llama-server or setting OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 in the official Ollama environment configuration cuts a 32B model's 32K KV cache from 8.00 GiB (FP16) down to 4.25 GiB with minimal perplexity drift, allowing a 24 GiB RTX 4090 or 32 GB Apple Silicon Mac to avoid CPU spillover.
For high-concurrency serving, vLLM PagedAttention (introduced in Kwon et al., SOSP 2023) eliminates contiguous KV fragmentation and supports FP8 KV cache quantization alongside 4-bit AWQ weight compression, though CUDA Graph capture reserves ~1.65 GiB+ upfront and enforces strict capacity checks without host-RAM spillover. On unified-memory workstations like the NVIDIA GB10 Grace Blackwell 128GB UMA platform (DGX Spark, ASUS Ascent GX10, and GIGABYTE AI TOP ATOM), ~118 GiB of coherent LPDDR5x is addressable after Linux kernel and page-cache reservations, making 70B Q4_K_M models at 64K-128K context capacity-safe while single-stream decode remains governed by the 273 GB/s memory bus (also see our GB10 nvidia-smi UMA telemetry fix).
Reference Matrix: Popular Local LLMs at 32K Context (Q4_K_M + Q8_0 KV)
| Model Architecture | Q4_K_M Weights | 32K KV (FP16 vs Q8_0) | Minimum Recommended Hardware |
|---|---|---|---|
| Llama 3.1 8B (32L, 8 KV) | 4.53 GiB | 4.00 GiB / 2.13 GiB | RTX 5070 Ti 16GB (~99 tok/s) or Mac mini M4 |
| Qwen 2.5 / R1-Distill 14B (48L, 8 KV) | 8.34 GiB | 6.00 GiB / 3.19 GiB | RTX 5070 Ti 16GB (12.64 GiB w/ Q8_0 KV) |
| Gemma 3 27B (62L, 16 KV) | 15.47 GiB | 15.50 GiB / 8.23 GiB | Mac mini M4 Pro 64GB, Dual 24GB, or GB10 128GB |
| Qwen 2.5 / R1-Distill 32B (64L, 8 KV) | 18.50 GiB | 8.00 GiB / 4.25 GiB | RTX 4090 24GB (Tight w/ Q8_0 KV), Mac 64GB, GB10 |
| Llama 3.3 70B / R1-70B (80L, 8 KV) | 39.87 GiB | 10.00 GiB / 5.31 GiB | NVIDIA GB10 128GB UMA, Mac 64GB (sysctl), Dual 24GB |
Local LLM VRAM, KV Cache & Unified Memory FAQ
Why does Ollama spill onto CPU when OLLAMA_NUM_PARALLEL is greater than 1?
Ollama pre-allocates the configured context window for every parallel slot (ctx * slots). Setting OLLAMA_NUM_PARALLEL=4 at 32768 context multiplies KV cache RAM by 4x (for example, 17.00 GiB at Q8_0 or 32.00 GiB at FP16 on Qwen 32B), pushing a 24 GB GPU into partial PCIe layer offload.
How much of 128 GB unified memory is usable for LLMs on NVIDIA GB10 systems?
On NVIDIA GB10 Grace Blackwell workstations (DGX Spark, ASUS Ascent GX10, GIGABYTE AI TOP ATOM), CPU and GPU share 128 GiB of LPDDR5x over NVLink-C2C. Leaving roughly 8 to 10 GiB for the Ubuntu ARM64 kernel, Docker daemon, and filesystem page cache yields ~118 GiB of safe zero-swap UMA capacity.
Why is single-stream LLM token generation bound by memory bandwidth rather than TOPS?
At batch size 1, generating each new token requires streaming the entire quantized weight matrix from DRAM into SRAM compute units. Consequently, maximum single-stream decode speed scales linearly with bandwidth_GBs / weight_decimal_GB, typically achieving 68% to 75% bus efficiency after runtime and attention overhead.
Companion Infrastructure Calculators & GB10 Benchmarks
Size RAM-safe shared_buffers, work_mem, and PgBouncer pools for n8n and RAG vector stacks.
Generate unbuffered proxy_buffering off blocks for long-running Ollama, vLLM, and MCP SSE streams.
Convert docker run --gpus all LLM commands into hardened Compose v2 YAML with loopback ports.
Compute macOS WindowServer 2x HiDPI Unified Memory swapchain overhead alongside local Metal LLMs.