Pick your tool
Three tools dominate local AI right now, and for a first run, one of them is the clear starting point:
- Ollama: the most widely recommended starting point (what it actually is, if you want the short version before installing). Runs from a terminal, handles model downloads and quantization automatically, and exposes a simple local API other tools can connect to.
- LM Studio: a visual, point-and-click app. Good if you'd rather browse and click than type terminal commands.
- llama.cpp: the engine underneath most of the above. Worth knowing about, not usually where a beginner starts directly.
This guide uses Ollama, since it's the fastest path for a first run and works the same way on Windows, Mac, and Linux. A full comparison of all three is planned for this site.
Step 1: Check your hardware first
Before downloading anything, know roughly what your machine can handle. Run your available RAM or VRAM through the hardware calculator, or if you just want a starting point: 8GB of RAM or VRAM comfortably fits a small model (around 7-8B parameters at Q4), which is exactly what this guide downloads first.
Step 2: Install Ollama
Download the installer for your operating system from Ollama's official site and run it like any other application. On Windows and Mac, this installs a background service plus a command-line tool. On Linux, it typically installs via a one-line shell command from Ollama's own instructions. On Windows specifically, install order matters for GPU detection; see running Ollama on Windows if your GPU doesn't get picked up.
Once installed, open a terminal (Command Prompt or PowerShell on Windows, Terminal on Mac and Linux) and confirm it's working:
If that prints a version number, you're ready for the next step.
Step 3: Download and run your first model
Ollama downloads and runs a model with a single command. For a first run on modest hardware:
The first time you run this, it downloads the model (several gigabytes, so it takes a few minutes depending on your connection). Every time after that, it starts almost immediately since the model is already on disk. This picks Llama 3.1 8B specifically because it's broadly supported and a manageable download; see picking a different Llama version once you're past this first run.
Step 4: Talk to it
Once the download finishes, you'll land in an interactive prompt right in the terminal. Type a message and press Enter. The model responds directly in your terminal window, entirely offline after the initial download; nothing you type gets sent anywhere. That offline behavior is worth understanding properly if you want to rely on it away from a connection, since a few things are worth checking before you actually need it. Type /bye to exit when you're done.
Troubleshooting common issues
- It's extremely slow: the model is probably too large for your hardware and is spilling into system RAM or swap. Try a smaller model or a more aggressive quantization; the calculator can tell you what actually fits.
- Out of memory errors: same root cause as above. Close other memory-heavy applications, or step down to a smaller model.
- The response quality feels weak: try a different model in the same size class before assuming your hardware is the limit. Some models are simply stronger than others at the same size; see the best local LLM by hardware tier guide for picks. If the question you're asking is broad or open-ended, that's a separate, real limit worth knowing about; see what local AI can't do well.