Ask an engineering team what they run in production and you will rarely hear the name of the model that topped last month's leaderboard. You hear about a mid-sized model, quantised, pinned to a version, wrapped in three layers of caching, and doing one narrow job extremely well. The gap between what gets written about and what gets deployed has never been wider.
That gap exists because the two things are optimised for different scoreboards. Frontier models are judged on capability ceilings. Production models are judged on cost per resolved request, and those two numbers pull in opposite directions once you are serving anything at volume.
The unit economics nobody puts in the blog post
A support automation feature handling 400,000 conversations a month has a budget. If each conversation averages six turns and each turn burns 2,000 tokens of context, you are moving roughly 4.8 billion tokens monthly before you have written a single line of business logic. At frontier pricing that is a line item your CFO will want a meeting about. At small-model pricing it is a rounding error.
The instinct is to reach for the cheapest model and accept worse output. That is the wrong move, and it is why so many cost-cutting exercises quietly get reverted two quarters later. The better move is to stop treating the model as a single decision.
Routing beats picking
Most request volume in a real product is boring. Classification, extraction, reformatting, short answers with a retrieved document already in context. A small model handles those at parity. The interesting 5% is where you spend: ambiguous intent, multi-step reasoning, anything where being wrong is expensive.
type Tier = "small" | "large";
function route(task: Task): Tier {
if (task.hasRetrievedContext && task.expectedTokens < 400) return "small";
if (task.requiresMultiStepReasoning) return "large";
if (task.errorCost > 50) return "large";
return "small";
}The router itself does not need to be clever. In most systems we have looked at, a handful of deterministic rules over metadata you already have captures the majority of the benefit. Teams that reach for a learned router first usually spend a month building it and land within a few points of the rules-based version.
What actually got smaller
Three things converged. Distillation stopped being lossy in ways users notice for narrow tasks. Quantisation to 4-bit went from a research curiosity to a default with acceptable degradation. And context handling improved enough that a small model with good retrieval frequently outperforms a large model working from memory.
- Distilled task-specific models now land within a few points of their teachers on the narrow slice they were trained for.
- Quantisation moved from lossy experiment to production default, cutting memory footprint enough to change what hardware you need.
- Retrieval quality improved faster than model size, so a well-fed small model beats a hungry large one more often than people expect.
- Serving stacks got better at batching, which changed the cost curve at exactly the volumes where it mattered.
We cut inference spend by 71% and our resolution rate went up. The model was never the bottleneck. Our retrieval was.
Where this goes next
The likely end state is not that small models win. It is that the question stops being interesting, in the same way nobody asks which database you use before asking what you are storing. Model choice becomes a per-endpoint decision made by whoever owns that endpoint, tracked on a dashboard next to latency and error rate.
Teams that get there early build the routing layer before they need it. Teams that get there late rewrite three services under budget pressure. It is not a hard call which is the better position.