Local LLM Hardware Buying Checklist: 10 Questions to Answer Before You Buy

Direct answer: Choose local LLM hardware from the workload outward. Name the exact model, quantization format, context length, concurrency target, and software stack before comparing GPUs. A larger advertised memory number is useful only when the machine can run your framework, keep enough headroom for the KV cache, and sustain the latency, noise, power, and privacy requirements of the actual workload.

This checklist is for planning and purchase evaluation. It is not a benchmark, and AI Compute Scout has not physically tested every system described here. For the memory arithmetic behind the first decision, start with How much GPU memory local LLMs need.

1. Define the job before the machine

“Run a local LLM” can mean a private chat assistant, code completion, document retrieval, batch summarization, fine-tuning, or an API serving several people. Those jobs have different bottlenecks.

Write down four targets:

Target	Example decision	Why it changes the hardware
Model	A specific 8B, 14B, 32B, or 70B checkpoint	Parameter count sets the starting weight footprint
Precision	FP16, INT8, 4-bit, or another format	Quantization changes memory, kernels, and output quality
Context	8K, 32K, or longer	KV-cache memory grows with context and active requests
Service level	One user, batch jobs, or concurrent API users	Concurrency changes cache pressure and throughput

If these answers are unknown, buying the most expensive workstation is premature. Rent a representative GPU or test on an existing machine first, then use measured peak memory and latency to narrow the purchase.

2. Calculate weights, then reserve headroom

The first estimate is:

weight memory ≈ parameter count × bytes per parameter

FP16 is roughly 2 bytes per parameter, INT8 about 1 byte, and 4-bit weights about 0.5 bytes. These are planning figures rather than guarantees. Quantization metadata, padding, embeddings, loader behavior, activations, and temporary buffers all add overhead. Hugging Face documents 8-bit and 4-bit loading as memory-saving techniques, not as universal compatibility promises.

Use the weights-only result to reject obviously small devices, not to declare a device ready. A 32B model in 4-bit format may have about 16 GB of weights, yet a 16 GB GPU leaves no useful room for cache or runtime buffers. The VRAM guide gives a capacity worksheet and explains why 20–30% spare capacity is only a starting heuristic for a simple single-user setup.

3. Treat the KV cache as part of the model

During generation, the runtime stores key and value tensors for the prompt and generated sequence. NVIDIA’s TensorRT-LLM documentation lists weights, activations, I/O tensors, and the KV cache among the major contributors to inference memory. The cache grows with context length and with the number of concurrent requests.

Ask the vendor or framework:

Does the runtime allocate for the maximum context at startup?
Can the KV cache use a different precision from the weights?
Does paged or shared caching change the concurrency limit?
What happens when the cache reaches the memory limit: rejection, offload, or slowdown?

For a personal assistant with short prompts, a smaller GPU may be comfortable. For a retrieval system with long documents or a team API, cache capacity can matter more than the weight estimate.

4. Check the software path, not only the silicon

Local AI workflows often combine a model loader, an inference server, GPU drivers, kernels, tokenizers, vector databases, and container tooling. Confirm all of them on the operating system and CPU architecture you intend to use.

NVIDIA’s local systems illustrate the point. DGX Spark uses a Grace Arm CPU and coherent unified memory, while a conventional desktop may use an x86 CPU and a discrete GPU. Both can expose CUDA-compatible workflows, but an adjacent native dependency or prebuilt binary may support only one architecture. Treat an Arm64 dependency audit as a purchase requirement, not a post-purchase surprise.

For every critical package, record:

Supported operating systems and CPU architectures.
Required driver and CUDA or runtime versions.
Supported quantization formats and GPU kernels.
Whether the framework can spill to system memory or another GPU.
How upgrades are tested and rolled back.

5. Understand unified memory versus discrete VRAM

Discrete GPUs present a dedicated accelerator-memory ceiling. Unified-memory systems expose a shared pool that the CPU and GPU can access coherently. The latter can make larger models possible, but it does not make all of that memory equivalent to high-bandwidth HBM or GDDR VRAM.

NVIDIA lists DGX Spark with 128 GB of coherent unified memory. That capacity can be valuable when model size is the binding constraint, but bandwidth, CPU participation, Arm64 compatibility, thermals, and the portion of memory available to the runtime still matter. A unified pool is a different architecture, not a simple “128 GB graphics card.” Compare it with DGX Station, the DGX Spark versus DGX Station analysis, and the three-way Spark, Station, and cloud comparison at the system level.

6. Size storage for the workflow, not just the checkpoint

Model weights are only one storage item. Plan for multiple quantizations, tokenizer files, container layers, embeddings, document indexes, logs, checkpoints, and a recovery copy. Leave free space for updates; a disk that is full can turn a routine model upgrade into an outage.

Ask whether the system supports:

A fast local NVMe drive for active models and cache.
Separate capacity for datasets and backups.
Encryption at rest and secure erase.
A repeatable way to rebuild the environment after a drive failure.

If data must stay local, include backup policy and access controls in the privacy decision. “Runs offline” is useful only when the surrounding storage and network paths are also controlled.

7. Budget for power, cooling, and noise

An always-on workstation has an operating cost and a physical footprint. Check the circuit, ventilation, room temperature, acoustic tolerance, and expected duty cycle. A system that is fast for a ten-minute demo may be uncomfortable beside a desk or expensive under a 24-hour serving load.

Use the vendor’s power figures as design inputs, not as a promise of your electricity bill. Measure wall power for the representative workload if the machine will run continuously. Also verify whether the system throttles under sustained load and whether replacement fans, filters, or support are available in your region.

8. Match performance to the user experience

Peak FLOPS do not directly answer whether an assistant feels responsive. Measure at least:

Time to first token.
Sustained tokens per second.
Peak accelerator and system memory.
Latency at the intended context length.
Throughput with the intended concurrency.
Output quality on representative prompts.

Run the exact checkpoint and serving stack you plan to use. A quantized model can fit but produce an unacceptable answer, and a model that is fast for one prompt can become slow when several users share the cache.

9. Decide whether privacy requires local ownership

Local hardware can simplify data locality and predictable access, but it also makes the owner responsible for patching, backups, identity controls, and incident response. A managed cloud GPU may provide better isolation or operational tooling for a small team, depending on the provider and contract.

Write down the data classification and the failure plan. If sensitive data cannot leave a controlled environment, verify telemetry, update channels, remote support, and outbound network behavior rather than assuming “local” means “air-gapped.”

10. Keep a cloud fallback for uncertain demand

Cloud GPUs are useful for bursty jobs, larger experiments, and pre-purchase validation. The hourly rate is only one input: add storage, idle behavior, egress, availability, setup time, and the work completed per hour. The DGX Spark versus cloud GPU comparison and local LLM hardware FAQ explain why a simple purchase-price-divided-by-hourly-rate calculation can mislead.

Before committing to a workstation, run a short pilot on the exact model and context. Record the instance type, software versions, peak memory, latency, and total bill. If the demand is intermittent, renting may remain the more economical and flexible choice even when the hardware can technically fit the model.

A one-page purchase checklist

Use this list in a quote or internal review:

Exact model checkpoint and license recorded.
Precision or quantization format selected and tested.
Weights-only estimate calculated.
Context length and maximum output defined.
Concurrency and latency target defined.
KV-cache, activation, and runtime headroom reserved.
Driver, framework, kernel, and CPU-architecture support verified.
Storage, backup, encryption, and recovery plan documented.
Power, cooling, noise, and duty cycle approved.
Cloud pilot or rollback option available.

Bottom line

The right local LLM machine is the smallest system that meets the measured workload with sustainable headroom and an operational plan. Start with the model and user experience, validate the software path, then compare discrete VRAM, unified memory, storage, power, and ownership costs. If the workload is still changing, keep a cloud GPU fallback and delay an irreversible hardware purchase.

This is an independent buyer’s guide based on the linked technical documentation. Hardware specifications, software support, and cloud prices change; verify the exact configuration and terms before purchasing.