Direct answer: Start with the model’s parameter count, precision, context length, and concurrency—not the model name alone. A rough weights-only estimate is parameters × bytes per parameter, but a working inference setup also needs room for the KV cache, runtime buffers, activations, the operating system, and the model loader. A GPU that barely fits the weights can still fail at the first real prompt.
This guide gives planning ranges rather than promises. Quantization formats, model architecture, context length, framework, and offloading behavior can change the result. For a broader hardware decision, compare this guide with DGX Spark versus cloud GPUs and the DGX Spark buyer’s guide.
The weight-size calculation
The simplest estimate is:
weight memory ≈ parameter count × bytes per parameter
For planning, FP16 is about 2 bytes per parameter, INT8 is about 1 byte, and 4-bit weights are about 0.5 bytes. These are approximations: metadata, scales, padding, embeddings, and the framework’s layout add overhead. Hugging Face documents 8-bit and 4-bit loading as ways to reduce the memory required to load larger models, but quantization is not a guarantee that every workload will fit. See the Transformers quantization documentation before choosing a format.
| Model size | FP16 weights only | INT8 weights only | 4-bit weights only |
|---|---|---|---|
| 8B | about 16 GB | about 8 GB | about 4 GB |
| 14B | about 28 GB | about 14 GB | about 7 GB |
| 32B | about 64 GB | about 32 GB | about 16 GB |
| 70B | about 140 GB | about 70 GB | about 35 GB |
These numbers are arithmetic estimates, not vendor specifications. Leave practical headroom rather than selecting a card whose advertised capacity exactly equals the estimate. As a starting heuristic, 20–30% spare capacity is more comfortable for single-user inference, and longer contexts or multiple users may need substantially more. Measure the exact model and runtime when a purchase decision matters.
Why the model can fit but still run out of memory
Context length grows the KV cache
During generation, the runtime keeps key-value (KV) tensors so it does not recompute the entire context for every new token. NVIDIA’s TensorRT-LLM documentation identifies weights, activation tensors, and I/O tensors—including the KV cache—as the major contributors to inference memory. The KV cache grows with context and, in a service, with the number of concurrent requests. See the TensorRT-LLM memory guide and its KV cache documentation.
That means a 32B model in 4-bit format may have roughly 16 GB of weights, yet a 16 GB card is not a realistic target. A 24 GB card might work for a short, single-user context with a compatible runtime; a long context, large output limit, or several simultaneous requests can push it over the limit. Treat those as experiments to validate, not as a purchase guarantee.
Runtime buffers and activations are real memory
The loader needs memory for tokenization outputs, temporary activations, CUDA or Metal buffers, and the model’s own metadata. Tensor parallelism can divide weights across GPUs, while pipeline parallelism and runtime settings change the memory profile again. Two applications using the same checkpoint can therefore report different free-memory requirements.
Quantization trades memory for constraints
4-bit and 8-bit formats make larger models practical on smaller devices, but the exact format matters. GPTQ, AWQ, NF4, and other formats are not interchangeable, and some kernels support only particular GPU architectures. Quantization can also affect quality. Test representative prompts, latency, and output quality instead of treating the bit count as a score.
A useful starting point by GPU capacity
The following is a starting-point worksheet for one user and a moderate context, not a compatibility table.
| Available accelerator memory | Sensible first experiments | What to verify before buying |
|---|---|---|
| 8 GB | 7–8B at 4-bit, short context | Runtime overhead, context limit, and whether the model needs CPU offload |
| 12–16 GB | 7–14B at 4-bit; selected 8B models at higher precision | KV-cache growth, batch size, and GPU kernel support |
| 24 GB | 14B at 4-bit with more headroom; selected 32B tests | Long-context behavior and whether the loader reserves memory up front |
| 48 GB | 32B at 4-bit more comfortably; selected 70B tests | Quantization quality, context, and multi-user concurrency |
| 80 GB or more | Larger 70B-class experiments and higher precision variants | Multi-GPU topology, throughput targets, and serving framework |
The word “sensible” is intentional. A model can technically load and still be too slow, too constrained, or too fragile for daily work. Start with the model and workload you actually need, then measure tokens per second, time to first token, peak memory, and quality.
Where unified memory changes the calculation
Some systems share a large pool of CPU and GPU memory instead of presenting only a discrete GPU VRAM number. NVIDIA lists DGX Spark with 128 GB of coherent unified memory. That can make large local models possible, but it does not turn every model into a high-throughput data-center deployment. Shared bandwidth, Arm64 software compatibility, thermals, and the portion of memory available to the accelerator still matter. The DGX Spark comparison and DGX Spark versus Mac Studio analysis cover those system-level trade-offs.
Local hardware or a cloud GPU?
Local inference is attractive when data should stay on a controlled device, the workload is steady, and low setup friction matters. A cloud GPU is often a better first step when demand is uncertain, jobs arrive in bursts, or the model needs a larger accelerator than the local budget can support. Our DGX Spark versus cloud GPU framework explains why hourly price is not the same as equivalent performance.
For a short experiment, renting capacity lets you test the exact checkpoint, context length, and serving stack before buying hardware. If you use a cloud marketplace, record the instance type, storage charges, idle behavior, and data-transfer terms alongside the GPU price. Do not compare a single advertised hourly number with a hardware purchase without accounting for the work completed per hour.
A pre-purchase VRAM checklist
- Name the exact model checkpoint and quantization format.
- Calculate the weights-only estimate from the parameter count.
- Set the target context length and maximum output tokens.
- Decide whether one request or several requests must run concurrently.
- Check the framework’s supported GPU architecture and operating system.
- Reserve memory for KV cache, activations, runtime buffers, and the host system.
- Test a representative prompt set and record peak memory and latency.
- Check storage, power, cooling, and noise for an always-on local system.
- Compare the measured result with a cloud instance if demand is uncertain.
- Re-test after model, driver, or framework updates.
Common mistakes
Treating advertised VRAM as usable model memory
The full capacity is not an invitation to allocate every byte to weights. The operating system, runtime, and cache need room, and a service may reserve memory for its maximum configured sequence length.
Comparing parameter count without precision
An 8B FP16 checkpoint and an 8B 4-bit checkpoint have the same parameter count but very different storage and runtime needs. Always state the precision and format beside the model name.
Ignoring concurrency
A desktop experiment with one prompt is not the same workload as an API serving ten users. Concurrency multiplies cache pressure and changes the acceptable latency target.
Assuming CPU offload is free
Offloading can make a model load when GPU memory is limited, but traffic across the CPU–GPU boundary can reduce throughput and increase latency. It is a useful escape hatch, not a substitute for sizing the target workload.
Bottom line
For a first local experiment, buy or borrow enough memory for the weights plus headroom, then validate the exact context and serving behavior. Quantization can make a model fit, but it cannot remove the need to test quality and latency. If the workload is still a hypothesis or occasionally needs a much larger model, use a cloud GPU first and keep the purchase decision open.
This article is a buyer’s guide based on the linked technical documentation. It does not claim physical testing of every GPU, model, or runtime. Hardware capacities, software support, and cloud prices change; verify the exact configuration before purchasing.