When Elias unboxed his high-compute graphics card with 16 GB of memory, he expected his local 32B coding model to fly. Instead, generation crawl-walked at two tokens per second. The card was blazing fast on paper, but the model weights breached the 16 GB threshold and spilled over into system RAM. The memory penalty was brutal: execution dropped anywhere between five to twenty times slower than fully on-device execution. Elias optimized for compute teraflops on the box rather than the single metric that actually governs local inference: addressable memory.
If you are architecting a local LLM setup in 2026, the baseline rule is absolute: size your GPU for VRAM first, prioritize memory bandwidth second, and treat compute teraflops as a distant third priority. When model weights reside entirely in video memory, execution remains high-throughput. The moment an offload to system memory occurs, token generation drops below normal human reading speeds.
The Memory Hierarchy of Local Inference
A large language model must load its parameter weights entirely into accessible memory to run efficiently. When an NVIDIA GeForce RTX 5090 runs Llama 3.3 70B fully inside VRAM, it pushes over 40 tokens per second. Force that same setup to offload layers into standard system RAM, and output collapses down to 1 or 2 tokens per second.
+-------------------------------------------------------------+
| 1. VRAM Capacity |
| Determines which model parameter size fits on-device. |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| 2. Memory Bandwidth |
| Dictates token generation speed once weights are loaded. |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| 3. Compute (Teraflops) |
| Minor tuning factor for local inference throughput. |
+-------------------------------------------------------------+
- VRAM Capacity establishes the hard boundary for what models you can instantiate without triggering disastrous system memory spillover.
- Memory Bandwidth dictates your token generation rate once the weights are comfortably resident in memory.
- Compute Teraflops represents the least critical specification for pure inference. Prioritize the memory bus and capacity, then treat compute as an afterthought.
Quantization: Compressing the Weights
Two years ago, a 70B parameter model required dedicated data center silicon. Today, quantization makes it viable on desktop workstations. Quantization compresses model weights from 16-bit precision down to 8-bit or 4-bit representations, slashing memory consumption by 50 to 75 percent.
- FP16: Full-quality baseline (140 GB footprint for a 70B model).
- 8-bit: Retains roughly 99 percent of full-precision output quality.
- 4-bit: Retains 90 to 95 percent of original quality, making it nearly indistinguishable from full-precision weights in day-to-day execution while compressing a 70B model down to roughly 40 GB.
The industry relies on two primary formats: GGUF (optimized for llama.cpp, targeting CPU or lightweight GPU inference as an accessible entry point) and safetensors (geared for GPU execution through vLLM). Architectures like Blackwell integrate native FP4 support, cutting memory requirements in half compared to 8-bit baselines. For local deployments, Q4_K_M within GGUF serves as the standard default.
Sizing VRAM by Model Class
Calculating memory requirements requires accounting for weight storage alongside the expanding KV cache, which grows based on prompt context length and concurrent requests.
| Model Size | Estimated VRAM Required (4-bit) | Representative Models | Deployment Target |
|---|---|---|---|
| 7-14B | 5-9 GB | Llama 3.1 8B, Qwen3 14B, Mistral 7B | 16 GB GPUs |
| 27-32B | 17-20 GB | Qwen 32B, DeepSeek-R1 32B | Sweet spot for 24 GB GPUs |
| 70B | ~40 GB | Llama 3.3 70B | Single 48 GB+ GPU or Dual 32 GB Cards |
| 109B (MoE) | ~70-75 GB | Llama 4 Scout | Single 80 GB+ Data Center Silicon |
| 405B | ~250 GB | Frontier open-weights | 256 GB+ Mac Studio or Multi-GPU Data Center Nodes |
The 27B to 32B class running on a 24 GB card represents the absolute sweet spot for price-to-performance in local AI. Qwen 32B and DeepSeek-R1 32B deliver competitive coding and reasoning capabilities while fitting inside a single 24 GB card at 4-bit precision with enough headroom for context. Scaling to 70B models hits a hard hardware barrier, demanding a single 48 GB-plus card or dual 32 GB configurations.
Ranked GPU Breakdown for 2026
Evaluating hardware across price brackets reveals how architectures handle distinct model tiers. Current pricing reflects mid-2026 market realities, where memory supply constraints have elevated cards above initial MSRPs. Check live listings and MillionMiner for verified availability.
| GPU Model | VRAM Capacity | Memory Bandwidth | Architecture & Features | Estimated Price | Optimal Target Workload |
|---|---|---|---|---|---|
| Intel Arc B580 | 12 GB GDDR6 | Unspecified | Battlemage | ~$250 | 7-8B Q4 (Budget starter) |
| RTX 5060 Ti | 16 GB GDDR7 | ~448 GB/s | Blackwell (FP4) | ~$450 | 7-14B Q4/Q8 (Entry workstation) |
| Radeon RX 7900 XTX | 24 GB GDDR6 | 960 GB/s | RDNA 3 (ROCm 7.x) | ~$800 | Up to 32B Q4 on Linux |
| RTX 3090 (Used) | 24 GB GDDR6X | 936 GB/s | Ampere | ~$900 | Up to 32B Q4 (Best value) |
| Mac Studio (M-series) | Up to 512 GB Unified | ~400-820 GB/s | Apple Silicon | From ~$2,000 | 70B to 405B Quantized |
| RTX 4090 | 24 GB GDDR6X | 1,008 GB/s | Ada Lovelace | ~$2,400-$3,500 | Max speed for 24 GB models |
| RTX 5090 | 32 GB GDDR7 | 1,792 GB/s | Blackwell (FP4) | ~$3,000-$5,000 | 32B with context, 70B pairs |
| RTX PRO 6000 Blackwell | 96 GB GDDR7 ECC | 1,792 GB/s | Blackwell (FP4) | ~$8,500 (Card) | Single-card 70B FP8/Q8 serving |
| A100 | 80 GB HBM2e | ~2 TB/s | Ampere (NVLink) | Enterprise | 70B Q4, QLoRA fine-tuning |
| H100 | 80 GB HBM3 | 3.35 TB/s | Hopper (FP8, NVLink) | Enterprise | High-concurrency 70B production |
| H200 | 141 GB HBM3e | 4.8 TB/s | Hopper | Enterprise | Memory-bound, long-context serving |
| B200 | 192 GB HBM3e | ~8 TB/s | Blackwell (FP4) | Enterprise | Frontier training and inference |
1. Intel Arc B580 12 GB
- Specs: 12 GB GDDR6, Battlemage architecture, priced around $250.
- Target Models: 7B to 8B models on Q4 (Llama 3.2, Phi-4 Mini, Gemma).
- Engineering Verdict: The absolute minimum cost of entry for local inference. It delivers strong price-to-performance for hobbyists testing pipelines, though the 12 GB ceiling restricts long-term utility.
2. NVIDIA RTX 5060 Ti 16 GB
- Specs: 16 GB GDDR7, ~448 GB/s bandwidth, Blackwell architecture with native FP4, priced around $450.
- Target Models: 7B to 14B models on Q4 or Q8 (Llama 3.1 8B, Qwen3 14B, Phi-4, Gemma 3, Mistral 7B) at 50 to 110 tokens per second.
- Engineering Verdict: The cheapest hardware offering native FP4 support. However, 16 GB is a short-term runway as base model parameters continue to swell.
3. AMD Radeon RX 7900 XTX 24 GB
- Specs: 24 GB GDDR6, 960 GB/s bandwidth, RDNA 3, ROCm 7.x, priced around $800.
- Target Models: Models up to 32B at Q4 via Linux, llama.cpp, and Ollama.
- Engineering Verdict: ROCm 7.x brings genuine stability to inference on Linux, offering 24 GB of memory hundreds of dollars below NVIDIA alternatives. The tradeoff is software ecosystem lock-in: NVIDIA remains the default for vLLM, Unsloth, and enterprise production tooling.
4. NVIDIA RTX 3090 24 GB (Used)
- Specs: 24 GB GDDR6X, 936 GB/s bandwidth, Ampere architecture, priced around $900 on the used market.
- Target Models: Up to 32B class at Q4 (Qwen 32B, scoring ~77 percent on SWE-bench, and DeepSeek-R1 32B), pulling roughly 87 percent of the RTX 4090 throughput.
- Engineering Verdict: The undisputed value champion for local AI. Four years post-launch, 24 GB provides a solid multi-year runway, making it the most cost-effective investment for solo developers.
5. Apple Mac Studio (M-Series)
- Specs: Up to 512 GB Unified Memory, ~400 to 820 GB/s bandwidth, starting around $2,000.
- Target Models: A 128 GB configuration runs Llama 3.3 70B at Q4 around 12 to 15 tokens per second. Higher 256 GB+ configurations host quantized 405B parameters.
- Engineering Verdict: Unified memory allows system RAM to act directly as VRAM, giving unmatched memory density and silent, low-power operation. The compromise lies in lower bandwidth compared to dedicated GPUs and exclusion from standard CUDA stacks.
6. NVIDIA RTX 4090 24 GB
- Specs: 24 GB GDDR6X, 1,008 GB/s bandwidth, Ada Lovelace architecture (discontinued October 2024), priced between $2,400 and $3,500.
- Target Models: 32B models at Q4, achieving roughly 120 tokens per second on 8B parameters in VRAM.
- Engineering Verdict: The fastest single 24 GB card available, but a questionable financial investment when priced near $3,000. It handles the same model footprints as the used 3090 at more than double the cost.
7. NVIDIA RTX 5090 32 GB
- Specs: 32 GB GDDR7, 1,792 GB/s bandwidth, Blackwell architecture with FP4, priced between $3,000 and $5,000.
- Target Models: 32B models with extensive context headroom, 34B models at Q8, and 45+ tokens per second on 70B models when fully accommodated. Its quantized throughput frequently outpaces older 40 GB A100 accelerators.
- Engineering Verdict: The consumer flagship of 2026. Pairing two 5090 cards yields 64 GB of VRAM, sufficient to process 70B at Q4.
8. NVIDIA RTX PRO 6000 Blackwell 96 GB
- Specs: 96 GB GDDR7 ECC, 1,792 GB/s bandwidth, lacking consumer NVLink, priced around $8,500 for the standalone card (or ~$22,000 for a fully configured workstation).
- Target Models: Single-card 70B models at FP8 or Q8 with heavy context, 109B MoE models (Llama 4 Scout) at Q4, pushing ~829 tokens per second on 32B models in vLLM and over 1,500 tokens per second on 120B 4-bit models.
- Engineering Verdict: Eliminates multi-GPU clustering issues by running 70B deployments on a single card for always-on team infrastructure.
9. NVIDIA A100 80 GB
- Specs: 80 GB HBM2e, ~2 TB/s bandwidth, NVLink support, data center form factor.
- Target Models: Single-card 70B at Q4, QLoRA fine-tuning of Llama 4 Scout (~70 to 75 GB in Unsloth), dense 7B to 34B models.
- Engineering Verdict: The premier cost-effective workhorse for hosting environments requiring true data center memory and clean NVLink scaling.
10. NVIDIA H100 80 GB
- Specs: 80 GB HBM3, 3.35 TB/s bandwidth, NVLink/NVSwitch, FP8 support, data center form factor.
- Target Models: Concurrent 70B serving via vLLM. While single-stream 70B clocks roughly 25 tokens per second, dozens of concurrent requests share the card without proportional degradation.
- Engineering Verdict: The production benchmark for high-concurrency serving pipelines.
11. NVIDIA H200 141 GB
- Specs: 141 GB HBM3e, 4.8 TB/s bandwidth, Hopper architecture.
- Target Models: 70B+ parameters at FP16 on a single GPU with deep KV cache allocations.
- Engineering Verdict: Retains Hopper compute while resolving memory-bound bottlenecks for production deployments serving extensive context windows.
12. NVIDIA B200 192 GB
- Specs: 192 GB HBM3e, ~8 TB/s bandwidth, Blackwell FP4 architecture. MillionMiner also provides the lower-power Blackwell B100 sibling with the identical 192 GB capacity.
- Target Models: Frontier training and inference, delivering 4 to 5 times the inference throughput of the H100 (up to 15 times on optimized LLM pipelines), and serving 70B at FP16 with massive KV caches. Native FP4 doubles effective throughput.
- Engineering Verdict: The pinnacle of density, maximizing throughput per GPU while minimizing physical node complexity.
Multi-GPU Interconnects: The NVLink Reality
Adding a secondary GPU increases raw VRAM capacity, which helps memory-bound inference tasks. However, NVIDIA removed physical NVLink bridges from consumer cards after the RTX 3090.
Consumer Multi-GPU (RTX 5090 / PRO 6000):
[ GPU 1 ] <====== PCIe Bus (Scaling Penalty) ======> [ GPU 2 ]
Result: ~70% to 80% real-world scaling (~1.4x - 1.5x throughput)
Enterprise Multi-GPU (A100 / H100 / B200):
[ GPU 1 ] <====== NVLink / NVSwitch ==============> [ GPU 2 ]
Result: ~85% to 93% clean throughput scaling
Pairing two RTX 5090s or two RTX PRO 6000s routes inter-card communication entirely over the PCIe bus. This hardware limitation caps real scaling around 70 to 80 percent, meaning dual cards produce 1.4 to 1.5 times the throughput rather than doubling it.
Whenever a model fits on a single card, choose the larger single GPU over split configurations to bypass the PCIe bottleneck. This reality makes a single 96 GB RTX PRO 6000 far more efficient than dual 32 GB cards for 70B deployments. Enterprise hardware utilizing NVLink scales at 85 to 93 percent efficiency, justifying its data center price premium.
Hardware Acquisition: The Utilization Equation
Hardware ownership comes down to continuous utilization metrics:
[ Continuous Hardware Utilization ]
|
+------------------------+------------------------+
| |
< 70% Utilization > 80% Utilization
| |
v v
[ Cloud GPU Rental ] [ Hardware Ownership ]
(Lower 3-year aggregate cost) (Breakeven in 4 to 12 months)
|
+------------+------------+
| |
v v
[ On-Premises ] [ Colocation ]
(Local noise, heat, (Data center power,
power, maintenance) rack cooling, SLA)
- Under 70% continuous utilization: Renting on-demand cloud capacity proves cheaper over a 3-year timeline while eliminating maintenance overhead.
- Over 80% continuous utilization: Direct ownership breaks even within 4 to 12 months for steady-state workloads like active inference endpoints, internal agents, or non-stop fine-tuning.
Running multi-GPU hardware on-premise requires managing heat output, power draws, noise, and driver stability. Colocation or dedicated AI GPU hosting resolves these physical constraints, providing ownership economics alongside enterprise power delivery, cooling infrastructure, and rack management.
Strategic Decision Matrix
- Hobbyists and Prototyping: Deploy the Intel Arc B580 or RTX 5060 Ti for 7B to 14B models. Step up to a used RTX 3090 for 32B headroom.
- Solo Engineers: Source a used RTX 3090 24 GB for peak value, or an RTX 5090 32 GB for top single-card speeds.
- Local 70B Inference: Pair two RTX 5090 cards (64 GB) at Q4, run a single RTX PRO 6000 96 GB to eliminate PCIe penalties, or utilize a 128 GB+ Mac Studio if you value silent operation over raw token output.
- Internal Team Serving: Implement a hosted single RTX PRO 6000 96 GB or an A100 80 GB node.
- Enterprise High-Concurrency Infrastructure: Deploy clusters of H100, H200, or B200 GPUs in dedicated hosting facilities via vLLM.
Elias eventually returned his 16 GB card, picked up a used 24 GB RTX 3090, and ran his 32B model entirely within local VRAM at full speed for less money. Identify your target model, map the required VRAM with sufficient KV cache headroom, and base your hardware choices on memory capacity and bandwidth rather than packaging teraflops.
Technical FAQ
What is the best all-around GPU for local LLMs in 2026?
For budget-conscious developers, the used 24 GB RTX 3090 (~$900) remains the best value, handling up to 32B models on a single card. The RTX 5090 (32 GB) is the fastest consumer option, while the RTX PRO 6000 Blackwell (96 GB) represents the best single-card option for 70B models.
How much VRAM is needed to run a 70B model locally?
A 70B parameter model at 4-bit precision requires roughly 40 GB of VRAM solely for its weights, excluding KV cache expansion. It demands a single 48 GB+ GPU (such as an A100 or RTX 6000 Ada), a dual RTX 5090 setup (64 GB combined), a single RTX PRO 6000 96 GB, or a 128 GB Unified Memory Mac Studio. Running 70B at unquantized FP16 requires roughly 140 GB of enterprise memory.
Which GPUs run Qwen 32B or DeepSeek-R1 32B?
At Q4_K_M quantization, 32B models consume roughly 19 GB of memory. They run cleanly on any 24 GB graphics card, including the RTX 3090, RTX 4090, RTX 5090, or AMD Radeon RX 7900 XTX, leaving sufficient space for context processing.
Is the RTX 5090 better than a used RTX 3090 for local inference?
The RTX 5090 provides significantly higher memory bandwidth (1,792 GB/s vs 936 GB/s) and an extra 8 GB of VRAM, but costs between $3,000 and $5,000. The used RTX 3090 sits around $900 and runs the same 32B model tier at roughly 87 percent of the 4090 generation's pace, making the 3090 vastly superior in capability per dollar.
Does quantization noticeably degrade output quality?
Loss is minimal for general tasks. Compared to the FP16 baseline, 8-bit precision preserves roughly 99 percent of original fidelity, while 4-bit quantization retains between 90 and 95 percent of model quality while decreasing memory demands by 50 to 75 percent. Q4_K_M via GGUF is the recommended operating standard for local inference.
Can local LLMs run on AMD or Intel graphics cards?
Yes. The AMD Radeon RX 7900 XTX (24 GB) with ROCm 7.x handles models up to 32B effectively on Linux via Ollama and llama.cpp. The Intel Arc B580 (12 GB) serves as a budget entry point for 7B to 8B models. However, NVIDIA remains the default standard for specialized environments like vLLM and Unsloth.
When should I buy hardware versus renting cloud GPUs?
If your continuous hardware utilization is below 70 percent, on-demand cloud rentals are more economical over a three-year timeframe. When sustained usage exceeds 80 percent, owning hardware hits the financial breakeven point within 4 to 12 months.
Can an Apple Mac Studio replace a dedicated GPU?
Yes. Apple Silicon uses unified memory accessible by both the CPU and GPU. A 128 GB Mac Studio runs a 4-bit 70B model at roughly 12 to 15 tokens per second, while 256 GB+ units can load quantized 405B models. The tradeoff is lower token output compared to high-bandwidth discrete GPUs and separation from standard CUDA toolchains.
What is the best data center GPU for production inference?
The NVIDIA H100 (80 GB) is the enterprise benchmark for multi-user concurrent serving using vLLM. The H200 (141 GB) expands capacity for deep context, and the B200 (192 GB) maximizes inference throughput at 4 to 5 times the capability of the H100. The A100 (80 GB) remains the preferred value tier for hosted fine-tuning and inference.