Direct answer: Local LLM hardware should be sized from the model weights plus KV cache, runtime overhead, context, and concurrency. Quantization can reduce memory, but it does not remove the need to test quality and latency. If the workload is uncertain or occasionally needs a much larger accelerator, rent a cloud GPU before committing to a local purchase.
This FAQ is a practical companion to the GPU memory guide and the local LLM hardware buying checklist. It describes planning principles, not guaranteed compatibility tables.
1. How much VRAM does an LLM need?
Start with parameter count × bytes per parameter for the weights. FP16 is roughly two bytes per parameter, INT8 roughly one byte, and 4-bit roughly half a byte. Then add metadata, runtime buffers, activations, KV cache, the operating system, and spare capacity. A 70B model at 4-bit may have roughly 35 GB of weights, but a 40 GB card is not automatically comfortable once context and runtime overhead are included.
2. Is 4-bit quantization always the best choice?
No. Four-bit formats often make a larger model fit on a smaller accelerator, but formats and kernels differ. GPTQ, AWQ, NF4, and other approaches can have different quality, speed, and hardware support. Hugging Face documents quantized loading as a memory-saving technique, not as a universal performance guarantee. Test representative prompts and compare quality before making 4-bit the default.
3. Why does a model fit at startup and then run out of memory?
The loader may report only weight memory. During generation, the runtime allocates KV cache, temporary activations, CUDA or Metal buffers, and I/O tensors. The cache grows with context length and, for a service, with the number of concurrent requests. NVIDIA’s TensorRT-LLM memory documentation treats these components as part of the real inference budget. Leave headroom instead of allocating the advertised capacity to weights alone.
4. How does context length affect memory?
Longer prompts and output limits increase the amount of key-value state retained for attention. The exact growth depends on model architecture, layers, head dimensions, cache precision, and runtime settings. A short one-user test can therefore succeed while a long-context API fails. Set a realistic maximum context and output length before choosing hardware, then measure peak memory under that setting.
5. Does concurrency multiply the requirement?
Usually, yes. Each active request can add KV-cache pressure and scheduling overhead. Ten users sending short prompts are not equivalent to one user sending one prompt. Measure one, two, and several simultaneous requests if the system will be shared. If concurrency is low and latency is flexible, a larger memory pool may be more valuable than peak compute. If concurrency is high, throughput and batching behavior become equally important.
6. Is unified memory the same as VRAM?
No. A unified system exposes CPU and GPU access to a shared memory pool, while a discrete GPU presents a dedicated high-bandwidth VRAM pool. NVIDIA lists DGX Spark with 128 GB of coherent unified memory. That can help a model fit, but bandwidth, CPU participation, runtime placement, and Arm64 compatibility still matter. Treat the architecture as a different trade-off, not as a 128 GB discrete graphics card.
7. Can CPU offload solve a VRAM shortage?
It can help a model load, but it often increases latency and reduces throughput because data crosses the CPU–GPU boundary. Offload is useful for experimentation or low-throughput private work. It is not a substitute for sizing the target service. Compare a GPU-only run with the offloaded run using the same prompt, context, and concurrency.
8. What is a reasonable memory starting point?
Use these as first experiments for one user and moderate context:
| Accelerator memory | Starting experiments | Main caveat |
|---|---|---|
| 8 GB | 7–8B at 4-bit | Short context and careful runtime settings |
| 12–16 GB | 7–14B at 4-bit | Cache and kernel support can be the limit |
| 24 GB | 14B at 4-bit; selected 32B tests | Leave room for long context and buffers |
| 48 GB | 32B at 4-bit; selected 70B tests | Quality, cache, and concurrency still require testing |
| 80 GB+ | Larger 70B-class or higher-precision experiments | Topology and serving throughput become central |
These are planning ranges, not promises. A model can technically load and still be too slow or unstable for daily use.
9. Does a larger parameter count always mean better output?
No. Quality depends on training data, instruction tuning, quantization, prompt design, retrieval, and the task. A smaller model with a good context and reliable tools can outperform a larger model that barely fits and constantly swaps memory. Choose a representative evaluation set before buying hardware for a parameter-count target.
10. How much storage should I plan for?
Keep room for the model files, multiple quantization variants, tokenizer data, container images, caches, logs, datasets, and outputs. A single checkpoint can be only part of the footprint. If you keep old versions for rollback, storage grows quickly. NVIDIA’s current DGX Spark hardware overview lists a 1 TB or 4 TB NVMe configuration; the usable space is lower after the operating system and working data.
11. Does a local setup automatically protect private data?
No. Local execution can reduce data movement, but the owner still controls accounts, disk encryption, network access, patching, backups, logs, and physical access. A shared workstation needs identity and authorization boundaries. Cloud infrastructure can be governed well, while an unmanaged local machine can leak data. Evaluate the complete data path rather than treating “local” as a security label.
12. When should I use a cloud GPU instead?
Use cloud capacity when demand is uncertain, jobs arrive in bursts, a project needs a different accelerator class, or you want to validate the exact model before buying hardware. A cloud marketplace can expose more options, but offer prices, storage, bandwidth, and availability vary. Record the selected offer and all billed resources. The DGX Spark versus cloud GPU analysis and three-way comparison cover that decision.
13. What should I measure in a proof of concept?
Measure time to first token, steady-state tokens per second, peak memory, prompt and output lengths, startup time, model load time, error rate, and quality on a fixed prompt set. For a service, add concurrent requests, queue time, and recovery after a restart. Record the driver, runtime, model revision, quantization, and hardware so the result is reproducible.
14. How do I compare local ownership with a cloud hourly rate?
Do not divide a purchase price by an advertised hourly number and call it break-even. Compare cost per completed useful job. For local hardware, include power, cooling, support, downtime, maintenance, and replacement risk. For cloud, include compute, storage, bandwidth, taxes, setup, idle time, and operator work. Use low, expected, and high utilization cases. The cloud GPU cost calculator provides a worksheet.
15. What software checks are easy to miss?
Check the architecture of Python wheels, container images, custom CUDA extensions, database clients, monitoring tools, compilers, and licensed binaries. Arm-based systems such as DGX Spark and DGX Station can be excellent NVIDIA development targets, but an x86-only dependency can block the workflow. Rebuild the exact image or run a short compatibility test before purchase.
16. Should I buy a GPU for one future model?
Usually not without a measured requirement. Model releases, quantization formats, runtimes, and prices change. Define a workload family—such as private retrieval, coding assistance, batch summarization, or image generation—then test representative models. A flexible cloud pilot or a modular workstation may be safer than buying for a single unverified parameter count.
17. How much headroom is enough?
There is no universal percentage, but a weights-only fit with zero spare capacity is a warning sign. Start with roughly 20–30% spare capacity for a simple single-user test, then add more for long context, multiple users, or a runtime that reserves memory up front. Validate the exact peak rather than relying on the heuristic when the purchase is material.
18. What is the simplest buying sequence?
- Name the model, precision, context, concurrency, and latency target.
- Estimate weights and leave room for cache and buffers.
- Audit architecture and runtime support.
- Run a reproducible local or cloud proof of concept.
- Measure useful throughput, stability, and total cost.
- Check power, cooling, storage, backups, and access control.
- Re-check prices and software support immediately before purchase.
OPINION: The best local LLM machine is the smallest system that meets the measured workload with comfortable headroom and an operating plan. If the workload is still a hypothesis, rent first and keep the purchase decision reversible.
Sources and verification note
Memory and quantization guidance was checked against Hugging Face Transformers quantization documentation, NVIDIA TensorRT-LLM memory documentation, and KV-cache documentation on August 31, 2026. DGX Spark capacity was checked against the official hardware overview, and cloud cost notes against Vast.ai pricing documentation on the same date. Verify the exact model, runtime, hardware configuration, and provider terms before purchase.