← Back

How to Reduce LLM Costs: Caching, Routing and Retries

TL;DR

Measure the cost of a completed task before changing your model. Compare caching, routing and retries without losing the quality controls that matter.

Conceptual illustration for How to Reduce LLM Costs: Caching, Routing and Retries

Start LLM cost optimisation with a completed business task: a correctly classified ticket, an accepted document extraction or a draft that meets your review rules. Measure what it costs to get that result. Then change one part of the system and check whether the saving survives the same quality test.

The cheapest model can produce the most expensive workflow. If it needs another attempt, sends more cases to a reviewer or quietly drops information, the saving on the API bill may disappear elsewhere.

This guide is for an application that already works well enough to measure. If you are still deciding what to build, start with a bounded AI proof of concept. Here, the job is to reduce the cost of delivering the same useful result, without removing the controls that make it dependable.

Find out which step is spending the money

Give each business task an identifier and group its model calls, retrieval requests, tool calls and retries under that identifier. A provider dashboard shows consumption; it usually needs your application context to explain why that consumption happened.

For each task, record the model and version, input and output tokens, cache reads and writes where applicable, number of attempts, tool charges, duration and final outcome. Keep the outcome definition stable. A response arriving without an API error does not prove that the task succeeded.

You do not need to copy every customer message into a logging platform to calculate this. Counts, timings, request identifiers and appropriately restricted evaluation records can answer different questions. Decide separately what content must be retained for diagnosis and who may see it.

Break the bill down by workflow and by step. If classification costs little but the application repeatedly summarises the same attachments, changing the classifier is unlikely to solve the main problem. Look at the expensive tail as well as the average: a few conversations with repeated tool failures can distort the total.

Decision sequence: measure the whole task, change one variable, test the same cases, then keep or revert.
Fictional calculation in the article; not provider pricing or a measured model benchmark.

If hosting is a material part of that cost, the LLM deployment guide compares model APIs, managed endpoints and self-managed serving using workload and operational responsibilities.

A worked example: cheaper calls, more repair work

The figures below are fictional planning assumptions, not provider prices, a Pixelfield client result or a measured model comparison. They show why the accounting boundary matters.

Assume 10,000 monthly tasks. Each starts with 2,000 input tokens and produces 300 output tokens. Use hypothetical rates of £2 per million input tokens and £8 per million output tokens. Assume each retry repeats the same token usage. Human repair costs £30 per hour and takes four minutes per referred task.

Monthly component Baseline Cheaper model scenario
Initial model calls £64.00 £32.00
Retry calls 1,000: £6.40 2,000: £6.40
Tasks requiring human repair 100 200
Human repair £200.00 £400.00
Selected model and repair subtotal £270.40 £438.40

For the second scenario, both token rates are halved. The retry and repair volumes are separate assumptions; a cheaper model does not inherently produce those outcomes. A task may be retried and then referred, so the categories can overlap without counting the repair twice.

The baseline initial charge is (10,000 × 2,000 × £2 + 10,000 × 300 × £8) / 1,000,000 = £64. Repair costs £2 per task. The API subtotal falls from £70.40 to £38.40, while the selected total rises by £168.

Hosting, retrieval, engineering and routine supervision are excluded from this illustration. Add them to your actual comparison. If some tasks remain unresolved after repair, record that too: dividing by 10,000 would then overstate the number of successful outcomes.

Remove work before changing the model

Inspect a representative trace from beginning to end. Does the workflow ask the model to do something your application already knows? A database lookup can return an account balance. A deterministic rule can check whether a required field is empty. Neither necessarily needs a generated explanation.

Repeated context deserves the same scrutiny. An entire conversation history may contain information the current step cannot use. A retrieval pipeline may return several copies of the same passage. Remove duplicates and irrelevant material, but retain the evidence needed for the decision. Test cases with exceptions are particularly useful here: shortening a policy document can accidentally remove the exception that makes the answer correct.

Output length is another practical target. If the next step consumes three fields, ask for those fields rather than a page of commentary. Set a realistic output limit and handle incomplete responses explicitly. A lower limit that creates truncated JSON and extra retries is a false saving.

Treat the different kinds of caching separately

Prompt caching reuses processing of a repeated prompt prefix. Response caching returns a previous answer. They solve different problems and need different checks.

Anthropic's prompt caching documentation describes cacheable prefixes, expiry and separate usage fields. Check the rules for your exact model and platform, including minimum length and write charges. A prompt that changes before the reusable prefix, or is reused too infrequently, may not deliver the saving you expect. Measure actual cache hits rather than assuming every request qualifies.

With response caching, ask whether the previous answer is still valid for this user, permission scope and data version. A public product description and a customer's current order status need very different expiry rules. Similar wording alone is insufficient evidence that two requests should share an answer.

Before deployment, test a permission change, a source update and an expired entry. The cache must not turn yesterday's authorised answer into today's disclosure or stale instruction. Include storage, lookup and invalidation work when calculating the benefit.

Route simpler tasks only when you can identify them

A smaller or less costly model can be a good candidate for a narrow task with clear acceptance criteria. Routing adds its own complexity: something must decide which path to use, and you must evaluate that decision as well as the models.

Begin with observable conditions such as supported document type, required fields and an allowed action. Avoid making an unvalidated confidence number the only reason a consequential request takes a cheaper path. Include difficult but plausible cases in the evaluation: conflicting fields, unusually long inputs and missing evidence.

Count the full fallback route. If a smaller model fails and the application sends the same task to a larger one, both calls belong to that task's cost. Compare this against starting with the larger model. Also check whether a failed first attempt changes the context of the second attempt in a way that affects quality.

Keep the initial release narrow. A successful extraction benchmark does not establish that the same routing policy works for negotiation, refunds or technical advice.

Stop retries becoming an invisible product feature

Repeated attempts need a reason, a limit and an owner. A transient network failure is different from a request the application cannot fulfil. Retrying an impossible extraction indefinitely consumes money without producing better evidence.

Set limits for attempts, elapsed time and tool activity. On reaching a limit, return an explicit unresolved state or send the task to a suitable queue. Do not replace a failed result with invented content simply to keep the apparent completion rate high.

When a tool changes another system, distinguish an uncertain response from a failed action. If a write succeeded but its acknowledgement was lost, repeating it may create a duplicate. Resolve the action's state before treating another model call as the answer. The AI integration guide covers the wider boundary between generated output and business systems.

Run a comparison you can roll back

Freeze a representative set of cases and their acceptance rules before changing the prompt, route or cache. Include ordinary work, known failures and cases where the right result is to abstain. Keep a separate set for later checks so that tuning does not become memorisation of your test pack.

Record these measures together:

  • Cost per accepted task, including fallbacks and measured review work.
  • Completion rate and reasons for unresolved cases.
  • Material errors, with separate gates for failures you cannot tolerate.
  • End-to-end latency, including time waiting for human review.
  • Cache hit rate, retry rate and the fraction sent down each route.

Choose pass criteria from the actual workflow. There is no universal acceptable accuracy percentage for both a suggested label and an executed payment. Compare like-for-like traffic, then release gradually with a way to restore the previous configuration. Save the prompt, model, retrieval and routing versions together so you can explain a change in results.

Decide whether the saving is worth maintaining

Every additional router or cache becomes something to monitor. Estimate the recurring saving alongside implementation and maintenance effort. If a change saves £20 a month but introduces a fragile dependency, leaving the simpler system in place may be the better decision.

Use the cost comparison worksheet to record your baseline, one proposed change, excluded costs and the evidence required to keep it. The worked example is filled in; replace its assumptions with measured usage and current contracted rates.

For a production application, bring one representative trace, the last billing period and a definition of an accepted result to an LLM development discussion. That gives the team something concrete to optimise: the cost of useful work, with its quality checks still intact.

Michal Vávra
Written by
Head of AI & Development

Michal has been with Pixelfield since September 2012 and has held technical leadership roles since 2017. He is the technical force behind our most demanding builds, combining AI expertise with the infrastructure and software architecture needed to make it work in production. His work spans LLM applications, multi-agent orchestration and complex integrations. He sets the engineering standards behind reliability, performance and scale, and guides the team through the decisions that determine how a system runs long after launch.