LAB-001/2026-10-04ACTIVE

LLM Inference Capacity Planner

Estimate the memory footprint of an LLM inference workload before you choose hardware. Configure the model, cache, and target system; the report updates immediately.

Configuration

Workload profile

Live estimate

00 / Model source

Hugging Face model

Model search?Searches public model repositories on the Hugging Face Hub. Selecting one imports available parameter count, context length, architecture, attention-head ratio, and quantization metadata. You can also paste a complete model URL.

HF Hub

01 / Runtime memory

Context & cache

The controls that most directly change inference memory after choosing a model and quantization.

Context length?The maximum number of input and generated tokens held for each sequence. Longer context increases KV-cache memory approximately linearly.Tokens

K cache precision?The precision used for cached attention keys. Lower precision reduces context memory but may affect attention accuracy and requires inference-engine support.V cache precision?The precision used for cached attention values. It can be configured separately from keys in supported runtimes; lower precision saves memory with a possible quality tradeoff.

Concurrent sequences?How many prompts or conversations the inference server processes at the same time. Each active sequence needs its own KV cache, so higher concurrency increases memory use.Slots

02 / Hardware target

Memory architecture

System type?Discrete GPUs have dedicated VRAM and separate host RAM. Unified-memory systems share one memory pool between the CPU and GPU.

GPU model?Search a curated catalog of NVIDIA, AMD, and Intel GPUs. Selecting a card fills its advertised memory capacity; runtime and software compatibility still vary.

52 options

GPU memory?Usable memory on each GPU. Select a GPU above to fill this automatically, or enter a custom board capacity.GiB / GPU

03 / Model metadata

Imported data & overrides

Populated from Hugging Face when available. Keep these controls for repositories with incomplete metadata or manual estimates.

Model parameters?The model's advertised parameter count, in billions. More parameters require more memory for weights. The slider snaps to common model sizes; enter any exact value in the box.Billion

KV heads vs query heads?The ratio of key/value heads to attention query heads. This is imported from the model config when available; GQA and MQA use fewer KV heads to reduce cache size.

Weight quantization?The numeric precision used to store model weights. An exact GGUF file selection overrides this generic estimate.

Capacity report

49.6 GiBMinimal VRAM required

324 GiB GPUs

Model weights: 36.7 GiB; Quantized parameters in memory
KV cache: 10.7 GiB; 32,768 tokens × 1 sequence × 84 attention layers
Compute buffers: 2.20 GiB; Allocator and inference workspace
Runtime subtotal: 49.6 GiB; Before 10% safety margin
Recommended VRAM: 54.5 GiB; Minimum required + 10% safety margin

Minimum host RAM47.4 GiB

On-disk model37.8 GiB

Run this configuration

llama.cpp server command

Uses your context, K/V cache precision, and concurrent sequences from above. Install a current llama.cpp build with the GPU backend for your hardware.

ShellModel source

GGUF model pathReplace the example path with your GGUF file matching the weight quantization above. For split files, use the first shard.

Vision projectorFlash Attention

Text-only mode skips the vision projector to save memory. Image input is disabled.

More launch settings

CPU threads (0 = auto)Prompt batch tokensMicro-batch tokensServer port

Prompt batch and micro-batch sizes control prompt processing, separate from parallel conversations. Smaller micro-batches can reduce workspace memory.

32,768 tokens/slot × 1 slot

llama-server \
  -m '/path/to/model.gguf' \
  -c 32768 \
  -np 1 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  -ngl all \
  --flash-attn on \
  --batch-size 512 \
  --ubatch-size 128 \
  --split-mode layer \
  --no-mmproj \
  --host 127.0.0.1 \
  --port 8080

Total context (-c) is 32,768 tokens; -np sets parallel slots. All model layers are requested on GPU. Layer splitting requires enough visible GPUs to meet the report. Cache quantization and Flash Attention support depend on the model and GPU backend; 16/8/4-bit selections use f16/q8_0/q4_0 respectively, and quantized cache storage includes block metadata. Open http://127.0.0.1:8080 after startup.

llama.cpp server flag reference

MODEL ASSUMPTION // Approx. 84 layers × 8,333 hidden size. Real usage varies by architecture, inference engine, tensor parallelism, and quantization format. Treat this as a planning estimate, not a compatibility guarantee.