What Counts as Small
The boundary is fuzzy and moving. Broadly, models under roughly 15 billion parameters are treated as small, with a growing class under 4 billion designed to run on phones and edge devices.
Capability at a given size has improved substantially. Small models today outperform models several times their size from two years ago, largely through better training data rather than architectural change.
Distillation
Distillation trains a smaller model to reproduce the behaviour of a larger one. The large model generates outputs, and the small model is trained to match them.
The result is a model that retains much of the teacher's capability on the distribution it was distilled over, at far lower cost. It does not retain general capability equally — a distilled model is strong on what it was distilled for and ordinary elsewhere.
This is the mechanism behind most of the small models that perform surprisingly well. It is also why benchmark performance can overstate their general usefulness.
Where Small Models Win
Narrow, repeated tasks. Classification, extraction, routing, sentiment, tagging. A small model doing one thing well beats a frontier model doing it at fifty times the cost.
High volume. When request counts are large, per-token cost dominates and small models change the economics entirely.
Latency-sensitive applications. Small models respond in a fraction of the time, which matters for anything interactive.
On-device. Privacy, offline capability and zero marginal cost, at the price of capability.
Where They Do Not
Complex multi-step reasoning. Long context. Open-ended generation requiring nuance. Anything where the range of possible inputs is wide and unpredictable.
Small models also fail less gracefully — they are more likely to produce confidently wrong output rather than acknowledging uncertainty.
The Practical Pattern
The architecture that works for most production systems is a cascade. A small model handles the majority of requests. A confidence check or a classifier routes the difficult minority to a larger model.
This typically cuts cost by a large multiple while preserving quality on the cases that matter, because most requests in most applications are not hard.
Choosing a Size by the Failure You Can Tolerate
Model selection is usually framed as a capability question and is better framed as a failure question: what happens when this gets it wrong, and how often can you afford that?
Small models fail differently from large ones. They are more likely to fail obviously — producing something malformed, off-topic or incomplete — which is easier to catch automatically. Large models fail more plausibly, producing fluent, confident, wrong output that passes a superficial review.
That changes the calculation. For a task where a wrong answer is caught downstream — a classification that gets validated, an extraction checked against a schema, a draft a human edits — a small model's higher error rate may cost almost nothing, and the latency and price advantage is real.
For a task where a wrong answer reaches someone and is believed, the plausible failure of a larger model is the more dangerous one, and neither size removes the need for the check.
The practical sequence: define the acceptable error rate first, measure both on your own data, and choose the cheapest one that clears the bar. Almost nobody does it in that order.
What Running One Actually Involves
The appeal of a small model is usually cost or control, and both come with work that the comparison tables omit.
Self-hosting is an operational commitment, not a one-off saving: serving infrastructure, capacity for peaks rather than averages, monitoring, and someone who can debug it at two in the morning. Below a certain volume, an API is cheaper once that is costed honestly.
The evaluation burden shifts to you. With a hosted frontier model you inherit the vendor's testing; with a small or distilled model on your task, you own the question of whether it is good enough, which means building an evaluation set before you deploy rather than after.
Updates are yours to manage. A hosted model improves without your involvement — and changes without your involvement, which cuts both ways. A self-hosted one stays exactly as it is until you do the work, which is a genuine advantage for reproducibility and a genuine cost in capability.
And on-device is a different discipline again: memory limits, battery, thermal throttling, and the fact that you cannot patch a model on a device the user has not updated.
The honest summary: small models win on latency, unit cost and data residency, and lose on the amount of engineering you have to supply yourself.