How to Reduce LLM Costs: Caching, Routing and Retries
TL;DR
Measure the cost of a completed task before changing your model. Compare caching, routing and retries without losing the quality controls that matter.
Start LLM cost optimisation with a completed business task: a correctly classified ticket, an accepted document extraction or a draft that meets your review rules. Measure what it costs to get that result. Then change one part of the system and check whether the saving survives the same quality test.
The cheapest model can produce the most expensive workflow. If it needs another attempt, sends more cases to a reviewer or quietly drops information, the saving on the API bill may disappear elsewhere.
This guide is for an application that already works well enough to measure. If you are still deciding what to build, start with a bounded AI proof of concept. Here, the job is to reduce the cost of delivering the same useful result, without removing the controls that make it dependable.
Find out which step is spending the money
Give each business task an identifier and group its model calls, retrieval requests, tool calls and retries under that identifier. A provider dashboard shows consumption; it usually needs your application context to explain why that consumption happened.
For each task, record the model and version, input and output tokens, cache reads and writes where applicable, number of attempts, tool charges, duration and final outcome. Keep the outcome definition stable. A response arriving without an API error does not prove that the task succeeded.
You do not need to copy every customer message into a logging platform to calculate this. Counts, timings, request identifiers and appropriately restricted evaluation records can answer different questions. Decide separately what content must be retained for diagnosis and who may see it.
Break the bill down by workflow and by step. If classification costs little but the application repeatedly summarises the same attachments, changing the classifier is unlikely to solve the main problem. Look at the expensive tail as well as the average: a few conversations with repeated tool failures can distort the total.

If hosting is a material part of that cost, the LLM deployment guide compares model APIs, managed endpoints and self-managed serving using workload and operational responsibilities.
A worked example: cheaper calls, more repair work
The figures below are fictional planning assumptions, not provider prices, a Pixelfield client result or a measured model comparison. They show why the accounting boundary matters.
Assume 10,000 monthly tasks. Each starts with 2,000 input tokens and produces 300 output tokens. Use hypothetical rates of £2 per million input tokens and £8 per million output tokens. Assume each retry repeats the same token usage. Human repair costs £30 per hour and takes four minutes per referred task.
| Monthly component | Baseline | Cheaper model scenario |
|---|---|---|
| Initial model calls | £64.00 | £32.00 |
| Retry calls | 1,000: £6.40 | 2,000: £6.40 |
| Tasks requiring human repair | 100 | 200 |
| Human repair | £200.00 | £400.00 |
| Selected model and repair subtotal | £270.40 | £438.40 |
For the second scenario, both token rates are halved. The retry and repair volumes are separate assumptions; a cheaper model does not inherently produce those outcomes. A task may be retried and then referred, so the categories can overlap without counting the repair twice.
The baseline initial charge is (10,000 × 2,000 × £2 + 10,000 × 300 × £8) / 1,000,000 = £64. Repair costs £2 per task. The API subtotal falls from £70.40 to £38.40, while the selected total rises by £168.
Hosting, retrieval, engineering and routine supervision are excluded from this illustration. Add them to your actual comparison. If some tasks remain unresolved after repair, record that too: dividing by 10,000 would then overstate the number of successful outcomes.
Remove work before changing the model
Inspect a representative trace from beginning to end. Does the workflow ask the model to do something your application already knows? A database lookup can return an account balance. A deterministic rule can check whether a required field is empty. Neither necessarily needs a generated explanation.
Repeated context deserves the same scrutiny. An entire conversation history may contain information the current step cannot use. A retrieval pipeline may return several copies of the same passage. Remove duplicates and irrelevant material, but retain the evidence needed for the decision. Test cases with exceptions are particularly useful here: shortening a policy document can accidentally remove the exception that makes the answer correct.
Output length is another practical target. If the next step consumes three fields, ask for those fields rather than a page of commentary. Set a realistic output limit and handle incomplete responses explicitly. A lower limit that creates truncated JSON and extra retries is a false saving.
Treat the different kinds of caching separately
Prompt caching reuses processing of a repeated prompt prefix. Response caching returns a previous answer. They solve different problems and need different checks.
Anthropic's prompt caching documentation describes cacheable prefixes, expiry and separate usage fields. Check the rules for your exact model and platform, including minimum length and write charges. A prompt that changes before the reusable prefix, or is reused too infrequently, may not deliver the saving you expect. Measure actual cache hits rather than assuming every request qualifies.
With response caching, ask whether the previous answer is still valid for this user, permission scope and data version. A public product description and a customer's current order status need very different expiry rules. Similar wording alone is insufficient evidence that two requests should share an answer.
Before deployment, test a permission change, a source update and an expired entry. The cache must not turn yesterday's authorised answer into today's disclosure or stale instruction. Include storage, lookup and invalidation work when calculating the benefit.
Route simpler tasks only when you can identify them
A smaller or less costly model can be a good candidate for a narrow task with clear acceptance criteria. Routing adds its own complexity: something must decide which path to use, and you must evaluate that decision as well as the models.
Begin with observable conditions such as supported document type, required fields and an allowed action. Avoid making an unvalidated confidence number the only reason a consequential request takes a cheaper path. Include difficult but plausible cases in the evaluation: conflicting fields, unusually long inputs and missing evidence.
Count the full fallback route. If a smaller model fails and the application sends the same task to a larger one, both calls belong to that task's cost. Compare this against starting with the larger model. Also check whether a failed first attempt changes the context of the second attempt in a way that affects quality.
Keep the initial release narrow. A successful extraction benchmark does not establish that the same routing policy works for negotiation, refunds or technical advice.
Stop retries becoming an invisible product feature
Repeated attempts need a reason, a limit and an owner. A transient network failure is different from a request the application cannot fulfil. Retrying an impossible extraction indefinitely consumes money without producing better evidence.
Set limits for attempts, elapsed time and tool activity. On reaching a limit, return an explicit unresolved state or send the task to a suitable queue. Do not replace a failed result with invented content simply to keep the apparent completion rate high.
When a tool changes another system, distinguish an uncertain response from a failed action. If a write succeeded but its acknowledgement was lost, repeating it may create a duplicate. Resolve the action's state before treating another model call as the answer. The AI integration guide covers the wider boundary between generated output and business systems.
Run a comparison you can roll back
Freeze a representative set of cases and their acceptance rules before changing the prompt, route or cache. Include ordinary work, known failures and cases where the right result is to abstain. Keep a separate set for later checks so that tuning does not become memorisation of your test pack.
Record these measures together:
- Cost per accepted task, including fallbacks and measured review work.
- Completion rate and reasons for unresolved cases.
- Material errors, with separate gates for failures you cannot tolerate.
- End-to-end latency, including time waiting for human review.
- Cache hit rate, retry rate and the fraction sent down each route.
Choose pass criteria from the actual workflow. There is no universal acceptable accuracy percentage for both a suggested label and an executed payment. Compare like-for-like traffic, then release gradually with a way to restore the previous configuration. Save the prompt, model, retrieval and routing versions together so you can explain a change in results.
Decide whether the saving is worth maintaining
Every additional router or cache becomes something to monitor. Estimate the recurring saving alongside implementation and maintenance effort. If a change saves £20 a month but introduces a fragile dependency, leaving the simpler system in place may be the better decision.
Use the cost comparison worksheet to record your baseline, one proposed change, excluded costs and the evidence required to keep it. The worked example is filled in; replace its assumptions with measured usage and current contracted rates.
For a production application, bring one representative trace, the last billing period and a definition of an accepted result to an LLM development discussion. That gives the team something concrete to optimise: the cost of useful work, with its quality checks still intact.