Last updated on
8 engineering levers to cut LLM API costs
The fastest way to halve an LLM API bill isn’t a cheaper model. It’s paying attention to what you send on every request and how often you send it. Most bills are inflated by the same handful of habits: re-sending a giant system prompt uncached, routing trivial calls to a frontier model, letting the model ramble, and never measuring any of it.
Here are eight levers that actually move the number, roughly in order of return.
1. Cache your fixed prompt prefix
Prompt caching is the highest-return lever for most production apps. If your requests share a large fixed prefix — a system prompt, a set of few-shot examples, a retrieved document — you can cache that prefix so repeat requests read it at a fraction of the input price.
On Anthropic’s API, a cache read costs about 10% of the base input price. A busy endpoint that reuses the same prefix thousands of times a day barely pays for the prefix at all. See the Anthropic caching docs and OpenAI’s caching guide for the mechanics on each platform.
The catch: caching is a prefix match. One changed byte invalidates everything after it. Keep the volatile stuff — timestamps, request IDs, the user’s actual question — at the end of the prompt, after the stable content. A datetime.now() interpolated into your system prompt quietly defeats the whole thing.
2. Add a semantic cache for repeat queries
Prompt caching optimizes the input. A semantic cache optimizes the output by skipping the model entirely for questions you’ve already answered.
In support bots and RAG systems, users ask near-identical questions constantly. Store past queries and their responses in a vector database, then check each new query against them. If a new query is close enough to a cached one, return the stored answer instead of calling the model. Those requests cost nothing and come back instantly. Even a 20–30% hit rate on a high-traffic endpoint noticeably lowers the monthly bill.
3. Route each request to a right-size model
Sending every request to a flagship model is the most common source of waste. Model prices span a wide range, and a lot of production work — classification, extraction, routing, short rewrites — runs fine on a small, fast model at a fraction of the cost.
Anthropic’s lineup shows the spread (standard rates per million tokens — check the pricing page for current rates):
| Model | Input | Output |
|---|---|---|
| Claude Haiku 4.5 | $1 | $5 |
| Claude Sonnet 5 | $3 | $15 |
| Claude Opus 4.8 | $5 | $25 |
Those are the standard Sonnet 5 rates; through August 31, 2026 it’s on introductory pricing of $2 input / $10 output.
A model router classifies each request by difficulty and sends the easy majority to a cheap model, escalating to a frontier model only when the task needs it. Forcing a weak model to retry something it can’t do is its own kind of waste, so the routing logic matters. But for most apps, a large share of traffic never needed the expensive model in the first place. Our comparison of Claude models for developers goes deeper on which tier fits which job.
4. Trim the context you resend
The API is stateless, so every turn resends the whole conversation. In a long agentic loop that means you’re paying to reprocess the same history again and again, and the bill grows with the transcript.
Trim it:
- Summarize old turns instead of carrying them verbatim.
- Prune intermediate reasoning and bulky tool results once they’ve served their purpose.
- Retrieve, don’t dump. Don’t paste a whole document when a relevant excerpt answers the question. An embedding search over the top few chunks is cheaper than a 100-page context window, and it cuts latency and hallucinations too.
Providers now offer server-side help for this (context editing to clear stale blocks, compaction to summarize history as it fills the window), but the cheapest token is still the one you never send.
5. Cap output and shrink tool schemas
Output tokens cost far more than input, usually around five times as much on frontier models. That makes the response the most expensive part of a request, and the easiest to let run long.
- Set a real
max_tokensceiling so a runaway generation gets cut off rather than billed in full. - Ask for less. “Answer in one sentence” or “return only the JSON” spends fewer of the priciest tokens than a prompt that invites a preamble and a summary.
- Keep tool schemas lean. With function calling, load only the tool definitions the current step needs instead of passing your entire library on every call — those schemas are input tokens you pay for each turn.
6. Split big jobs into a pipeline
Teams often try to do everything in one enormous prompt: analyze, extract, summarize, generate SQL, write the report — all in a single call. Splitting the work into stages is usually both cheaper and better.
Break it into steps — classify, then extract, then generate — and let each step use the model it actually needs. Classification runs on a small model; the reasoning-heavy step goes to a frontier model; code generation goes to one of the models that are actually good at code. You stop paying frontier prices for the easy stages, and each prompt gets narrower and more reliable because it’s doing one job instead of five.
7. Batch anything that isn’t real-time
If a workload doesn’t need an immediate answer — nightly evals, bulk classification, generating embeddings over a backlog — send it through the batch API. Anthropic and OpenAI both price batch processing at 50% off standard rates in exchange for asynchronous delivery (typically within a few hours, up to 24). For offline jobs, that’s a flat halving of cost with no quality loss.
8. Measure real tokens before optimizing
You can’t optimize a number you’re guessing at. Instrument the basics — prompt and completion tokens, cost per request, cost per feature, cache hit rate, and which models handle which share of traffic — usually through an LLM gateway. That’s how you find the expensive prompts and the endpoints worth fixing first.
Measure with the provider’s own token-counting endpoint, not a generic library. Tokenizers are model-specific: a count from tiktoken (OpenAI’s tokenizer) applied to a Claude request can be off by 15–20% on plain English and more on code. Count against the exact model you’ll call, then you have a baseline to optimize against and a way to verify the cache is actually being hit.
Where to start
If you do only one thing, cache your fixed prefix. It’s a config change with the highest return. After that, route the easy traffic to a smaller model and cap your output. Those three cover most of the waste in a typical app. Semantic caching, context trimming, pipelines, and batching are the follow-ups once the obvious leaks are sealed.
None of this trades away quality. It trades away tokens you were paying for and not using.
Frequently asked questions
What's the single most impactful optimization for LLM API costs?
Prompt caching for inputs, and semantic caching for outputs. If you send the same large system prompt or document on every request, prefix caching means you pay roughly a tenth of the input price on repeat reads. For recurring user queries, a semantic cache skips the model entirely, so those requests cost nothing.
Are input and output tokens priced the same?
No. Output tokens cost far more — usually about five times the input rate on frontier models. That's why capping max_tokens, asking for terse structured output like JSON, and keeping tool schemas small all matter: a chatty model burns your budget on the priciest tokens.
Should I just always use the cheapest model?
No. Route by task. A small, fast model like Claude Haiku 4.5 handles classification, extraction, and routing well; a frontier model earns its price on hard reasoning and agentic work. Sending everything to the top tier is the most common waste, but forcing a weak model to retry a task it can't do is its own kind of waste.
How do I measure tokens accurately before I send a request?
Use the provider's own token-counting endpoint, not a generic library like tiktoken. Tokenizers are model-specific, and a mismatched one can undercount by 15–20% on plain English and much more on code — enough to overflow a context budget you thought had headroom.