What a Token Is
Models process text as tokens rather than words or characters. In English a token is roughly three-quarters of a word — around four characters. A thousand words is approximately 1,300 tokens.
Tokenisation is less efficient for other languages, for code, and for unusual formatting. The same meaning expressed in some languages costs meaningfully more tokens than in English, which matters for multilingual applications.
Input and Output Are Priced Differently
Output tokens typically cost several times more than input tokens, because generation requires a forward pass per token while input is processed in parallel.
The practical consequence: long prompts are cheaper than long answers. A system that reads a large document and returns a short summary is far cheaper than one that reads a short prompt and writes at length.
Instructing a model to be concise is a cost control, not only a quality preference.
Where Costs Hide
Conversation history. Every turn resends the whole conversation. A long chat costs progressively more per message, and this grows quadratically over a session.
RAG context. Retrieved chunks are input tokens. Retrieving ten large chunks per query multiplies input cost.
Reasoning tokens. Reasoning models generate substantial internal thinking that is billed as output even though the user never sees it.
Retries. Failed requests and automatic retries are billed.
Prompt Caching
Most providers now offer caching for repeated prompt prefixes. Where a long system prompt or document is sent identically across many requests, the cached portion is charged at a large discount.
Structuring prompts so the stable content comes first and the variable content last is what makes caching effective. It is one of the highest-return optimisations available and is frequently overlooked.
Controlling Spend
- Route by task. Use a small cheap model for classification and routing, reserve the expensive model for work that needs it.
- Cap output length explicitly rather than relying on the model to be brief.
- Trim conversation history — summarise older turns instead of resending them.
- Tune retrieval — fewer, better chunks beat many mediocre ones on both cost and quality.
- Batch where latency permits; batch APIs are typically discounted substantially.
- Monitor per-feature cost, not just total. Aggregate spend hides the one feature consuming most of it.
Where the Bill Comes From, and Why Estimates Are Wrong
Cost forecasts for a language-model feature are usually wrong in the same direction, because the estimate is built from the visible request and the bill is driven by what surrounds it.
The components people forget:
The system prompt is charged on every call. A long instruction block is a fixed cost per request, multiplied by volume. Halving it is frequently the largest single saving available and nobody looks there.
Conversation history is resent. Models are stateless, so a ten-turn conversation sends the preceding nine turns again on the tenth call. Cost grows with the square of conversation length unless you truncate or summarise.
Retrieved context dominates. Where documents are injected, the retrieved passages typically outweigh everything else, so retrieval precision is a cost decision as much as a quality one.
Retries and failures are billed. A malformed response that triggers a retry costs twice.
And reasoning output is billed as output, so a model that deliberates at length costs materially more than its headline price implies.
Reducing It Without Making the Product Worse
The levers, in rough order of saving against effort:
Route by task. Most requests in most applications are not hard. Sending everything to the most capable model is the single largest and most common overspend — classification, extraction and formatting rarely need the top tier.
Shorten the system prompt. Fixed cost, every call, and usually accumulated by addition over months without anyone removing anything.
Cache the stable prefix. Where the provider supports it, the unchanging front of a prompt can be reused across calls at reduced cost — the highest-return change for applications with a long fixed preamble, and it requires putting the stable part first, which is a code change rather than a setting.
Cap output length deliberately, and ask for the format you need rather than trimming prose afterwards.
Manage the conversation window: summarise old turns instead of resending them.
And instrument before optimising. Log tokens per request by feature and by user before changing anything, because the intuition about where the spend sits is wrong more often than it is right — and a cost programme that starts with guesses usually optimises the wrong call.