Why Run Locally
Privacy. Data never leaves the machine. For confidential documents, regulated industries, or organisations with strict data policies, this is frequently the deciding factor.
Cost predictability. No per-token charge. Once the hardware exists, volume is free.
Offline capability and no dependence on a vendor's uptime, rate limits or pricing changes.
Control. The model does not change under you. A frontier API model can behave differently next month; a local model file does not.
What You Can Actually Run
The constraint is memory. A rough guide for quantised models:
- 8GB RAM — small models around 7–8B parameters, usable for summarisation, classification and simple chat
- 16GB — comfortable 7–14B, adequate for most everyday tasks
- 32GB — 30B-class models, noticeably more capable
- 64GB+ or a dedicated GPU — 70B-class, approaching useful general capability
Apple Silicon machines punch above their weight because memory is shared with the GPU. A Mac with substantial unified memory runs models that would need an expensive discrete GPU on other hardware.
Quantisation
Quantisation reduces the numerical precision of model weights, shrinking memory requirements substantially with modest quality loss.
4-bit quantisation is the common practical choice — roughly a quarter of the memory of full precision, with degradation most people find acceptable. 8-bit loses less and saves less. Below 4-bit, quality falls off sharply.
This is what makes local AI viable on ordinary hardware at all.
The Tools
Ollama — the simplest route. One command downloads and runs a model, with an API compatible with common client libraries. The default recommendation.
LM Studio — a graphical application with model browsing and a chat interface. Easiest for non-technical users.
llama.cpp — the inference engine underneath much of the ecosystem. Maximum control, most effort.
The Honest Trade-Off
A model you can run on a laptop is meaningfully less capable than a frontier API model. It will be worse at complex reasoning, long context, code generation and instruction-following.
For summarising documents, drafting routine text, classification, extraction and simple conversation, the gap is often small enough not to matter. For hard problems it is large.
Many organisations end up with both: local models for high-volume routine work on sensitive data, API models for the tasks that genuinely need frontier capability.
What the Hardware Actually Decides
The constraint on running a model locally is memory before it is speed, and the arithmetic is simple enough to do before buying anything.
A model needs roughly its parameter count in memory, scaled by the precision of its weights. Full precision needs about two bytes per parameter; common quantisations bring that to roughly one byte, or a half, or less. So the same model can need several times more or less memory depending only on how it was compressed.
Then add the context. The working memory for the conversation grows with how much text is in play, and a long context can require as much again as the weights.
Unified memory changes the calculation. On machines where the processor and graphics share one pool, the usable ceiling is much higher than a discrete graphics card of similar price, which is why some laptops run models that desktops cannot.
The practical sequence: work out your memory ceiling, subtract what the operating system needs, and that determines the size and quantisation you can run. Everything else is a consequence of that number.
The Honest Trade, Stated Plainly
Local models are frequently recommended for the wrong reason and dismissed for the wrong reason.
The genuine advantages: data never leaves the machine, which for confidential or regulated material can be the entire argument; no per-token cost, so high-volume repetitive work is effectively free once the hardware exists; it works without a connection; and the model does not change underneath you.
The genuine costs: a capability gap against frontier hosted models that is real and, for hard reasoning tasks, large; slower generation on consumer hardware; setup and maintenance that is a hobby for some people and an obstacle for most; and quantisation degrading quality in ways that are not visible until a specific task exposes them.
Where it clearly wins: sensitive documents, bulk repetitive processing, experimentation without a bill, offline use, and learning how these systems behave.
Where it clearly does not: anything needing the current frontier of capability, anything where your time is worth more than the token cost, and any production system where somebody other than you depends on it staying up.
Most people who try it settle on both, and that is the correct answer rather than a failure to commit.