KV Cache Memory Estimator
Select a preset LLM or enter custom model parameters, along with precision, batch size, and sequence length, to estimate the GPU memory required for KV Cache, model weights, and activations, and check it against common GPUs.
FreeOnline Tool
Loading…
How to Use
- Choose a common large language model from the "Preset Model" dropdown (e.g. Llama 3 8B), or select "Custom Model..." to manually enter layer count, hidden dimension, intermediate size, vocabulary size, KV head count (for GQA), and attention head count.
- Select the inference "Precision" (FP16/FP8/FP32/INT8/INT4) — lower precision generally means lower memory usage.
- Enter "Batch Size" and "Sequence Length (tokens)" — both directly affect the size of the KV Cache memory footprint.
- Three cards at the top live-update "KV Cache Memory", "Model Weight Memory", and "Activation Memory"; the "Total Memory Requirement" card below sums all three.
- The "GPU Compatibility" table automatically checks whether common GPUs (H200/H100/A100 80GB/L40S/A100 40GB/RTX 4090/RTX 3090) have enough memory for the current total. The "Precision Comparison" table lists memory needs across all five precisions (FP16/FP8/FP32/INT8/INT4) for the same configuration, making the precision-vs-memory tradeoff easy to see. Click "Load Sample Data" to quickly load Llama 3 8B + FP16 + batch=1 + sequence length 4096.
Features
- Includes built-in preset parameters for common LLMs for quick selection, and also supports fully custom model architecture parameters (layers, hidden dimension, intermediate size, vocabulary size, GQA's KV head count, and attention head count).
- Supports switching between five inference precisions (FP16/FP8/FP32/INT8/INT4), calculating memory usage per the byte width of each.
- Estimates all three memory components at once — KV Cache, model weights, activations — and sums them into a total memory requirement.
- Includes a built-in memory-capacity table for seven mainstream GPUs (H200 141GB, H100 80GB, A100 80GB, L40S 48GB, A100 40GB, RTX 4090 24GB, RTX 3090 24GB), flagging whether each can fit the current configuration.
- Provides a five-precision memory comparison table for the same model configuration, showing at a glance how much memory dropping precision (e.g. FP16 to INT4) actually saves.
Use Cases
Choosing the right GPU before deploying an LLM
Before deciding which GPU to deploy a self-hosted LLM service on, an ML engineer enters the model's parameters and expected batch size/sequence length, using the GPU compatibility table to quickly judge whether an H100, A100, or a consumer card will suffice.
Evaluating whether quantization enables a cheaper GPU
A team considering quantizing a model from FP16 down to INT8 or INT4 to cut costs uses the precision comparison table to see the exact memory reduction, and evaluates whether a consumer card like the RTX 4090 could replace an A100.
Planning memory budget for long-context scenarios
For use cases requiring very long context (e.g. 32K or 128K tokens), increasing the sequence length parameter shows how fast KV Cache memory grows, letting teams decide ahead of time whether they need a bigger GPU or a KV Cache compression scheme.
Estimating total memory needs under concurrent users
For a multi-user online inference service, adjusting Batch Size to simulate concurrent requests estimates the memory ceiling at different concurrency levels, informing capacity planning.
FAQ
Why do I need to enter KV Head count and Attention Head count separately in "Custom Model"?
These two fields support GQA (Grouped Query Attention): when KV head count is smaller than attention head count, the model is using GQA to reduce KV Cache memory (multiple attention heads share one KV group). If the model uses standard MHA instead, just set both fields to the same value.
Where do the preset model parameters come from, and are they kept up to date?
The layer count, hidden dimension, and other structural parameters for preset models are fixed reference data built into the tool (based on each model's publicly known architecture) — it makes no network calls to fetch or refresh them. If a model gets a new version, the parameters may not be updated; check the official model card as the final source of truth.
How is "activation memory" estimated, and how accurate is it?
Activation memory is a rough estimate based on batch size, sequence length, and hidden dimension. Real inference frameworks (vLLM, TensorRT-LLM, etc.) each have their own memory management and optimizations (PagedAttention, memory reuse, etc.), so actual usage can be significantly lower than this estimate — treat it as a conservative upper bound only.
Does lower precision always mean worse model quality?
The tool only estimates memory — it doesn't evaluate the impact of precision on model quality. Aggressive quantization like INT4 can bring some quality loss, but the actual impact varies by model and quantization method. Combine this with real evaluation results rather than deciding based on memory numbers alone.
If GPU memory exactly equals the estimated total, does the table count it as "fits"?
The table's logic marks it as fitting whenever GPU memory ≥ the estimated total, but real deployments need extra headroom for the OS, other processes, and memory fragmentation. Choose a GPU with comfortable margin rather than one that's just barely enough.