The AI Feature Shipped. Then the Token Bill Arrived.

There is a moment every engineering team remembers. The AI feature is live. Users love it. Adoption curves look beautiful. And then the first real invoice from your LLM provider lands, and the number has an extra digit nobody budgeted for.

This is not a hypothetical scenario. It is the predictable second chapter of nearly every production AI deployment. The proof of concept ran on a few hundred calls a day against a flagship model. Now the feature handles thousands of requests, each one burning through tokens at the same per-call cost that seemed trivial during testing. The math that worked at demo scale breaks at production scale.

The instinct is usually one of two extremes: downgrade the model across the board and accept lower quality, or absorb the cost and hope volume discounts materialize. Neither is a real strategy. What actually works is more architectural than financial — it is the soil work of designing systems that route intelligently and cache deliberately so every token spent earns its keep.

Why LLM Costs Scale Differently Than You Expect

Traditional API costs tend to be roughly linear. Double the users, roughly double the compute. LLM costs behave differently because token consumption is shaped by prompt design, context window size, and the unpredictability of natural language input.

A single user query might cost ten times more than another depending on how much context the system needs to retrieve, how long the prompt template is, and how verbose the model's response turns out to be. Multiply that variance across thousands of daily requests and forecasting becomes guesswork.

Three cost drivers deserve attention before any optimization begins:

• Input tokens — the prompt, system instructions, and any retrieved context you send to the model. These are often the largest and most controllable cost factor.
• Output tokens — the model's response. Harder to control directly but shapeable through prompt engineering and response-format constraints.
• Model tier — the difference between a flagship model and a capable smaller model can be 10-50x per token. Not every task needs the most expensive option.

Understanding this breakdown is foundational. You cannot reduce LLM API costs effectively if you are optimizing the wrong variable.

Model Routing: The Right Model for the Right Task

Model routing is the practice of directing different requests to different models based on the complexity, risk, or nature of the task. It is one of the highest-leverage architectural decisions in any AI pipeline.

The core insight is simple: most production AI workloads are not uniformly complex. A system that summarizes support tickets, drafts email responses, classifies intent, and generates detailed technical reports is performing four fundamentally different tasks. Sending all four to the same flagship model is like hiring a principal engineer to reset passwords.

How to Design a Routing Layer

A well-engineered routing layer evaluates each incoming request and assigns it to the most cost-effective model that can handle it at the required quality level. This involves several design decisions:

Task classification. Before a request reaches any LLM, a lightweight classifier — sometimes rule-based, sometimes a small model itself — determines the task type. Classification, extraction, and simple summarization can often be handled by smaller, faster, less expensive models. Complex reasoning, nuanced generation, and tasks requiring deep domain knowledge route to more capable models.

Confidence-based escalation. A more sophisticated pattern routes requests to a smaller model first, evaluates the confidence or quality of the output, and escalates to a larger model only when the initial result falls below a threshold. This means you pay flagship prices only for the requests that genuinely need flagship capability.

Domain-specific fine-tuned models. For high-volume, narrowly scoped tasks, a fine-tuned smaller model often outperforms a general-purpose large model — at a fraction of the cost. The investment in fine-tuning pays for itself quickly when the task is well-defined and repetitive.

The Trade-offs Are Real

Model routing is not free. It adds architectural complexity. You are now maintaining relationships with multiple model providers or managing multiple model versions. Your testing surface area expands. Prompt templates may need to be adapted for different models since they respond differently to the same instructions.

The routing logic itself needs monitoring. If your classifier miscategorizes a complex request as simple and sends it to a lightweight model, the user sees degraded quality. Building observability into the routing layer — tracking which model handled each request, what the quality score was, and what the cost was — is not optional. It is the engineering discipline that makes routing sustainable.

Prompt Caching: Stop Paying for the Same Work Twice

If model routing is about choosing the right tool, prompt caching is about not repeating work you have already done. In many production systems, a surprising percentage of LLM calls are functionally identical or near-identical.

Consider a system that answers questions against a knowledge base using retrieval-augmented generation. The system prompt, the formatting instructions, and even many of the retrieved context chunks repeat across requests. Every repeated token is a cost you are paying again for no new value.

Layers of Caching

Exact-match response caching. The simplest form. If the same input has been seen before, return the cached output without making an API call at all. This works well for classification tasks, FAQ-style queries, and any workflow where inputs cluster around common patterns. The cache hit rate can be surprisingly high — we have seen workflows where 30-40% of requests are effective duplicates.

Semantic caching. A step beyond exact matching. Incoming queries are embedded and compared against cached queries using similarity search. If a new query is semantically close enough to a previously answered one, the cached response is returned. This requires careful threshold tuning — too aggressive and you serve stale or slightly wrong answers; too conservative and you rarely get cache hits.

Prefix caching and context reuse. Several major LLM providers now support server-side prefix caching, where repeated prompt prefixes — system instructions, static context, few-shot examples — are cached on the provider's infrastructure. You still pay for the unique portion of each request, but the shared prefix is processed at a significantly reduced cost. Designing your prompts to maximize the shared prefix length is a form of prompt engineering that directly impacts your bill.

What Caching Cannot Solve

Caching is powerful but it has boundaries. Highly personalized outputs, creative generation tasks, and conversations with long, evolving context windows benefit less from caching. Stale cache entries can serve outdated information if your underlying knowledge base changes and your cache invalidation strategy is not tight.

The discipline here is in knowing what to cache, how long to cache it, and when to invalidate. A cache with no expiration policy is a liability, not an optimization.

Prompt Engineering as Cost Engineering

This is the optimization most teams overlook because it does not feel like infrastructure work. But the design of your prompts is one of the most direct controls you have over token consumption.

Verbose system prompts with redundant instructions, overly detailed few-shot examples, and open-ended output formats all inflate costs on every single call. Tightening prompts — removing unnecessary context, constraining output format, using structured output schemas — reduces both input and output tokens without changing model capability.

A well-engineered prompt that produces reliable, brand-consistent behavior in 400 tokens is worth more than a sprawling 2,000-token prompt that achieves the same result with more waste. This is where prompt engineering and system design intersect with cost control: every token in your system prompt is multiplied by every request your system handles.

Building the Observability Layer

None of these strategies work without measurement. You need to know, per request:

• Which model handled it
• How many input and output tokens were consumed
• Whether a cache was hit
• What the quality of the output was (even if measured by proxy)
• What the total cost was

This telemetry is the foundation that makes ongoing optimization possible. Without it, you are guessing. With it, you can identify which workflows are the most expensive, which are the best candidates for routing to a smaller model, and where caching would have the highest impact.

The teams that build this observability from the start — as part of the AI pipeline architecture, not bolted on after the bill gets painful — are the ones that scale their AI features sustainably.

Where to Start: The Highest-Leverage Moves First

If your LLM costs are already uncomfortable or you are planning a feature that will scale to high volume, here is the order that tends to compound most quickly:

1. Audit your prompts. Measure token counts on your highest-volume workflows. Tighten system prompts, constrain output formats, eliminate redundant context. This is often a 20-40% cost reduction with zero infrastructure changes.
2. Implement exact-match caching on your most repetitive workflows. Even a simple hash-based cache with a reasonable TTL can cut costs meaningfully.
3. Introduce model routing for your clearest task-complexity tiers. Start with two tiers — a capable smaller model for straightforward tasks and your current model for complex ones. Expand from there as you gather data.
4. Design for prefix caching by restructuring prompts so the static portion comes first and the variable portion comes last. Take advantage of provider-side caching where available.
5. Build the observability layer that lets you see cost-per-request, quality-per-model, and cache hit rates in real time.

This is not a one-time project. It is an ongoing practice of measurement and refinement — the kind of automated, integrated system design that turns an AI feature from a cost center into a scalable asset.

The Real Cost of Not Optimizing

The risk is not just the bill itself. Uncontrolled LLM costs force hard choices downstream: capping usage, degrading features, or abandoning AI capabilities that users have come to rely on. The teams that treat cost optimization as an architectural concern from the beginning — as part of the foundational design, not a crisis response — are the ones whose AI features actually get to flourish at scale.

This is the work we do at Figtree Development. Not bolting AI onto existing systems and hoping the economics work out, but engineering automated pipelines with model routing, caching, prompt design, and observability built in from the ground up. The kind of soil work that lets the results compound instead of the costs.

If your AI feature is live and the token bill is growing faster than the value it delivers, that is a solvable problem — and it starts with understanding exactly where your tokens are going and which ones are earning their keep.

Book a free 20-minute discovery call with us at Figtree Development. We will map where your highest-leverage automation savings are hiding and build a plan to reduce costs without touching the quality your users depend on.