GEARFLOW LAB // LOCAL LLM VRAM & KV CACHE UTILITY 100% Client-Side JS · Zero Telemetry

Local LLM VRAM, KV Cache & Decode Bandwidth Calculator

Calculate exact Transformer weight footprint (GiB), Grouped-Query Attention (GQA) KV cache memory across parallel slots, runtime workspace overhead, single-stream Decode tok/s, and copy-ready CLI flags for Ollama, vLLM, and llama.cpp.

Verified Lab Profiles:

1. Model, Quantization & Runtime Controls

Transformer GQA Hyperparameters 262 KiB/tok (FP16 KV)
32,768 tok (32K)

2. Real-Time VRAM Budget, Decode tok/s & 6-Target Fit Matrix

100% GPU Fit on NVIDIA GB10 128GB UMA 23.86 / 118.0 GiB (20.2%)
All 64 layers (18.50 GiB), 4.25 GiB of Q8_0 KV cache (32,768 tokens x 1 slot), and 1.11 GiB of ollama buffers fit inside 118.0 GiB usable memory with 94.14 GiB spare headroom.
Total Required
23.86 GiB
25.62 GB decimal
Model Weights
18.50 GiB
32.76B @ 4.85 bpw
GQA KV Cache
4.25 GiB
136.0 KiB/tok · 1 slot
Decode Estimate
~9.2 tok/s
Max ceiling: 13.7 tok/s
Proportional Memory Stack vs Usable Hardware Ceiling 23.86 GiB / 118.0 GiB ceiling
Weights: 18.50 GiB KV Cache: 4.25 GiB Runtime Overhead: 1.11 GiB Single-stream bandwidth-bound @ 273 GB/s

Simultaneous 6-Hardware Target Comparison Matrix

Click any row to select target
Hardware Target Usable Ceiling Utilization Decode tok/s (1-Stream) Fit Verdict
RTX 5070 Ti 16GB 896 GB/s GDDR7 16.0 GiB 149.1% ~30.1 tok/s (max 45.1) OOM: Exceeds Capacity
RTX 3090 / 4090 24GB 1008 GB/s GDDR6X 24.0 GiB 99.4% ~33.9 tok/s (max 50.8) Tight Fit (OS/Display Headroom Warning)
Mac mini M4/M4 Pro 32GB 273 GB/s · 28 GiB wired max 24.0 GiB 99.4% ~9.2 tok/s (max 13.7) Tight Fit (OS/Display Headroom Warning)
Mac mini M4 Pro 64GB 273 GB/s · 56 GiB wired max 48.0 GiB 49.7% ~9.2 tok/s (max 13.7) 100% GPU Fit
NVIDIA GB10 128GB UMA GX10 / AI TOP ATOM / DGX Spark · 273 GB/s 118.0 GiB 20.2% ~9.2 tok/s (max 13.7) 100% GPU Fit
Dual GPU 48GB (2x 24GB TP=2) 1750 GB/s effective TP=2 48.0 GiB 49.7% ~58.8 tok/s (max 88.1) 100% GPU Fit

Production Launch CLI (Ollama systemd, vLLM & llama-server)

# Target: Qwen 2.5 / Qwen 3 32B (Q4_K_M, 32768 ctx, KV=q8_0, Slots=1) -> 23.86 GiB
# 1. Ollama CLI & systemd drop-in (/etc/systemd/system/ollama.service.d/override.conf)
# [Service]
# Environment="OLLAMA_FLASH_ATTENTION=1" "OLLAMA_KV_CACHE_TYPE=q8_0" "OLLAMA_NUM_PARALLEL=1" "OLLAMA_CONTEXT_LENGTH=32768"
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_NUM_PARALLEL=1 ollama run qwen2.5:32b

# 2. vLLM OpenAI-Compatible Server (PagedAttention + CUDA Graph)
vllm serve Qwen/Qwen2.5-32B-Instruct --max-model-len 32768 --max-num-seqs 1 --kv-cache-dtype fp8 --gpu-memory-utilization 0.90

# 3. llama.cpp Native HTTP Server (GGUF + Flash Attention)
llama-server -m ./models/Qwen2.5-32B-Instruct-Q4_K_M.gguf -ngl 99 -fa -ctk q8_0 -ctv q8_0 -c 32768 -np 1 --host 127.0.0.1 --port 8080

Exact Transformer VRAM & GQA KV Cache Equations

Autoregressive LLM memory splits into three distinct allocations: static quantized weights, dynamic Key-Value (KV) attention state, and runtime compute buffers. Unlike legacy Multi-Head Attention, modern architectures such as Llama 3.3 70B and Qwen 32B use Grouped-Query Attention (GQA) with 8 KV heads instead of 64 query heads, shrinking per-token cache footprint by 8x. For each active token across slots concurrent requests, KV memory equals 2 * layers * kv_heads * head_dim * bytes_per_element.

In llama.cpp K-quant formats, super-block scales add fractional overhead above nominal bit width (for example, Q4_K_M averages ~4.85 bpw and Q8_0 uses 34 bytes per 32-element block, or 8.50 bpw / 1.0625 B/elem). Passing -fa -ctk q8_0 -ctv q8_0 to llama-server or setting OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 in the official Ollama environment configuration cuts a 32B model's 32K KV cache from 8.00 GiB (FP16) down to 4.25 GiB with minimal perplexity drift, allowing a 24 GiB RTX 4090 or 32 GB Apple Silicon Mac to avoid CPU spillover.

For high-concurrency serving, vLLM PagedAttention (introduced in Kwon et al., SOSP 2023) eliminates contiguous KV fragmentation and supports FP8 KV cache quantization alongside 4-bit AWQ weight compression, though CUDA Graph capture reserves ~1.65 GiB+ upfront and enforces strict capacity checks without host-RAM spillover. On unified-memory workstations like the NVIDIA GB10 Grace Blackwell 128GB UMA platform (DGX Spark, ASUS Ascent GX10, and GIGABYTE AI TOP ATOM), ~118 GiB of coherent LPDDR5x is addressable after Linux kernel and page-cache reservations, making 70B Q4_K_M models at 64K-128K context capacity-safe while single-stream decode remains governed by the 273 GB/s memory bus (also see our GB10 nvidia-smi UMA telemetry fix).

Reference Matrix: Popular Local LLMs at 32K Context (Q4_K_M + Q8_0 KV)

Model Architecture Q4_K_M Weights 32K KV (FP16 vs Q8_0) Minimum Recommended Hardware
Llama 3.1 8B (32L, 8 KV) 4.53 GiB 4.00 GiB / 2.13 GiB RTX 5070 Ti 16GB (~99 tok/s) or Mac mini M4
Qwen 2.5 / R1-Distill 14B (48L, 8 KV) 8.34 GiB 6.00 GiB / 3.19 GiB RTX 5070 Ti 16GB (12.64 GiB w/ Q8_0 KV)
Gemma 3 27B (62L, 16 KV) 15.47 GiB 15.50 GiB / 8.23 GiB Mac mini M4 Pro 64GB, Dual 24GB, or GB10 128GB
Qwen 2.5 / R1-Distill 32B (64L, 8 KV) 18.50 GiB 8.00 GiB / 4.25 GiB RTX 4090 24GB (Tight w/ Q8_0 KV), Mac 64GB, GB10
Llama 3.3 70B / R1-70B (80L, 8 KV) 39.87 GiB 10.00 GiB / 5.31 GiB NVIDIA GB10 128GB UMA, Mac 64GB (sysctl), Dual 24GB

Local LLM VRAM, KV Cache & Unified Memory FAQ

Why does Ollama spill onto CPU when OLLAMA_NUM_PARALLEL is greater than 1?

Ollama pre-allocates the configured context window for every parallel slot (ctx * slots). Setting OLLAMA_NUM_PARALLEL=4 at 32768 context multiplies KV cache RAM by 4x (for example, 17.00 GiB at Q8_0 or 32.00 GiB at FP16 on Qwen 32B), pushing a 24 GB GPU into partial PCIe layer offload.

How much of 128 GB unified memory is usable for LLMs on NVIDIA GB10 systems?

On NVIDIA GB10 Grace Blackwell workstations (DGX Spark, ASUS Ascent GX10, GIGABYTE AI TOP ATOM), CPU and GPU share 128 GiB of LPDDR5x over NVLink-C2C. Leaving roughly 8 to 10 GiB for the Ubuntu ARM64 kernel, Docker daemon, and filesystem page cache yields ~118 GiB of safe zero-swap UMA capacity.

Why is single-stream LLM token generation bound by memory bandwidth rather than TOPS?

At batch size 1, generating each new token requires streaming the entire quantized weight matrix from DRAM into SRAM compute units. Consequently, maximum single-stream decode speed scales linearly with bandwidth_GBs / weight_decimal_GB, typically achieving 68% to 75% bus efficiency after runtime and attention overhead.

TOPIC CLUSTER HUB & COMPANION BROWSER UTILITIES

Companion Infrastructure Calculators & GB10 Benchmarks

PostgreSQL & PgBouncer Memory Calculator →

Size RAM-safe shared_buffers, work_mem, and PgBouncer pools for n8n and RAG vector stacks.

Nginx 502/504 & SSE Timeout Generator →

Generate unbuffered proxy_buffering off blocks for long-running Ollama, vLLM, and MCP SSE streams.

Docker Run to Compose & .env Converter →

Convert docker run --gpus all LLM commands into hardened Compose v2 YAML with loopback ports.

Mac & 5K2K Monitor PPI & HiDPI Calculator →

Compute macOS WindowServer 2x HiDPI Unified Memory swapchain overhead alongside local Metal LLMs.