What actually matters: VRAM first, then bandwidth, then RAM
Three numbers decide how a local model runs on your machine, and they matter in this order:
- VRAM (or unified memory) capacity decides whether a model fits at all. If the model's weights plus context don't fit in memory, it either won't load or it spills into much slower system RAM.
- Memory bandwidth decides how fast it runs once it fits. Two cards with the same VRAM can generate tokens at very different speeds if their bandwidth differs.
- System RAM matters far less for GPU inference. It mostly comes into play if you're offloading part of a model that doesn't fully fit in VRAM, or running CPU-only.
Use the hardware calculator to check a specific model and quantization against your own numbers before reading the tier breakdown below.
Best GPUs for local LLMs by budget tier
VRAM capacity is the deciding factor at every tier. Speed differences matter, but a card that doesn't have enough VRAM won't run the model at all, regardless of how fast it is. One thing worth knowing before you shop: 2026 has seen a broad, memory-driven price surge across nearly every tier here, so treat "budget" and "flagship" as relative positions in the lineup, not a promise about what you'll actually pay. Check current street prices before assuming any of these sit near their original launch price.
| GPU | VRAM | Comfortable at Q4 | Notes |
|---|---|---|---|
| RTX 3060 | 12 GB | Up to ~14B | Budget-tier by lineup position; NVIDIA relaunched this card in mid-2026 near its original price, but it has climbed noticeably since, so check current pricing rather than assuming it's still cheap |
| RTX 4060 Ti 16GB | 16 GB | Up to ~20B | Best VRAM-per-dollar in the budget-to-mid range |
| RTX 5070 | 12 GB | Up to ~14B | Faster than the 3060, same VRAM ceiling |
| RTX 5070 Ti / 5080 | 16 GB | Up to ~20B | Comfortable mid-high tier |
| RTX 4090 | 24 GB | Up to ~32B | Discontinued as new stock in 2026; still widely used and available used, at prices well above its old MSRP |
| RTX 5090 | 32 GB | Up to ~32B with headroom | Current flagship; 70B models are a tight Q4 fit at best. Street pricing has run well above its $1,999 MSRP through 2026, so confirm current cost before treating this as a default pick |
| RX 7900 XTX | 24 GB | Up to ~32B | Strong VRAM for the price, but weaker AI software support (ROCm) than NVIDIA's CUDA ecosystem for most local-AI tools |
"Comfortable at Q4" means the model's Q4_K_M weights plus typical short-context overhead fit with headroom, per the same math the hardware calculator uses. Run your exact model size through it rather than trusting the table alone. If you're on Windows and none of these GPUs seem to be getting used, check the install-order issue first before assuming it's a hardware problem.
Apple Silicon: how unified memory changes the math
Apple Silicon Macs don't have separate VRAM. The CPU, GPU, and Neural Engine share one pool of unified memory, so the Mac's total RAM figure is the number that matters for fitting a model, not a separate GPU spec.
- MacBook Air (M4): ships with 16GB unified memory, configurable to 24GB or 32GB. The 16GB base model comfortably handles 8B models at Q4; 24GB opens up the 13-14B range.
- Mac Studio (M5 Max): 36GB base, configurable up to 128GB, with up to 614GB/s memory bandwidth. Enough headroom for 32B-70B models depending on configuration.
- Mac Studio (M5 Ultra): 96GB base, configurable up to 512GB, with up to 1.2TB/s bandwidth. This is the tier where very large open-weight models become usable locally, at a significant price premium (Apple prices unified memory upgrades at roughly $25 per GB).
The tradeoff: unified memory bandwidth is generally lower than a high-end discrete GPU's dedicated VRAM bandwidth, so token generation speed on a Mac can lag a well-matched NVIDIA card even when both technically "fit" the same model. Capacity is Apple Silicon's advantage; raw speed is usually the discrete GPU's.
Buying used: what to actually check
A used GPU is often the best value in this hobby, since VRAM capacity (the number that matters most for local AI) doesn't degrade with age the way some other components do. Before buying:
- Confirm the VRAM capacity matches the exact model name. Some GPU families ship in more than one VRAM configuration under similar names; check the specific SKU, not just the family name.
- Ask about usage history. A card that ran 24/7 under sustained load (mining or render farms) has more wear than one used for gaming a few hours a day, even at the same age.
- Check the power connector generation. Newer high-wattage cards use different connector standards than older ones; make sure it matches your power supply without an adapter chain you don't trust.
- Verify it's still on a driver branch that gets updates. Older cards eventually age out of current driver support, which can affect compatibility with newer inference software.
When you don't need a GPU at all
Small models (roughly 7-8B parameters at Q4) run acceptably on CPU and system RAM alone, without a dedicated GPU. Expect noticeably slower generation, commonly in the range of a few tokens per second rather than the tens of tokens per second a GPU delivers, which is fine for occasional use or non-interactive batch tasks but frustrating for real-time chat. If you already own a machine with at least 16GB of system RAM and no interest in buying a GPU, it's worth trying a small model this way before spending anything.