TL;DR Every production AI feature has two budgets: money per request and time per request. Most of the engineering that happens after the prototype works is spending those two budgets wisely. The levers are the same short list every time, model choice, token count, caching, and how you stream.

A prototype gets to ignore both budgets. It runs once, for you, on your machine, and nobody is counting the seconds or the cents. A shipped product cannot ignore either. The moment real users arrive, every request costs money and makes someone wait, and those two numbers multiplied by your traffic decide whether the feature survives. The work of taking a demo to production is, more than anything, the work of getting cost and latency under control without giving up the quality that made the demo worth shipping.

Two budgets

Cost is driven by tokens. You pay for the input tokens you send and the output tokens the model generates, each multiplied by that model's price. As I covered in the tokens piece, you pay and wait per token, not per word, so a verbose prompt and a chatty answer both show up on the bill. Latency is driven by three things: the size of the model, the length of the output, and the number of round trips. A bigger model thinks more slowly per token, a longer answer takes more tokens to finish, and every extra call to the model adds a full network round trip on top. Keep these two budgets separate in your head, because a change that helps one can quietly hurt the other.

The cost levers

The first lever is model routing. Most requests are easy, so send them to a smaller, cheaper model and escalate to a frontier model only when the task genuinely needs it. A classifier or a cheap first pass in front of an expensive model often cuts the bill by an order of magnitude without users noticing. The second lever is prompt caching: if your system prompt and your reference context are stable across requests, cache them so you are not paying full price to resend the same tokens every call. The third lever is trimming input tokens through context engineering, sending the model only what the task needs rather than everything you have, which I go deeper on in the context engineering piece. The fourth is capping output length, since you pay for every token the model generates. And where the work can tolerate delay, batch requests to earn the cheaper batch rate.

If you want the intuition for where the wins are, it is almost always fewer tokens and a cheaper model. Those two dominate. The clever caching and batching tricks matter, but they are rounding errors next to routing an easy request away from a frontier model, or cutting a bloated prompt in half. Start with the two big levers before you reach for the small ones.

The latency levers

The most important latency lever is not making the model faster, it is changing what the user feels. Stream the tokens as they are generated, and the number that matters becomes time-to-first-token, not total time. An answer that takes eight seconds to finish feels fast if the first words appear in under a second. That is the gap between perceived latency and actual latency, and streaming is how you close it. After that, shorten the output, because a shorter answer finishes sooner. Then attack round trips: every agent step and every tool call is a full round trip back to the model, so a loop that takes five steps waits five times, which is why the loops-and-graphs and tool-calling pieces both matter here. Collapse sequential steps where you can, use a faster and smaller model where quality allows, and parallelize any calls that do not depend on each other rather than running them in a line.

Measure before you cut

Do not guess at any of this. Log tokens, cost, and latency on every request, then look at the numbers before you touch a line of code. And look at the p95, not just the average, because the average hides the slow tail that users actually complain about, and a healthy average with an ugly p95 is a feature that feels broken for one user in twenty. Once you can see the numbers, optimize the biggest line item first. Premature optimization here is pure wasted effort: shaving tokens off a prompt that is one percent of your spend, while a runaway retry loop eats the other ninety-nine, is motion without progress. The one rule that overrides all of this is that you never trade quality for cost or latency without an eval to catch the regression. As I argued in the evals piece, the cheaper model and the shorter answer are only wins if your eval confirms the output is still good enough. Otherwise you have just saved money on a feature that no longer works.

Why it matters

Whether a pilot becomes a real deployment usually does not come down to raw capability. The demo already proved the capability, that is why there is a pilot. It comes down to unit economics and responsiveness at scale: can this run at the price the business can absorb, and fast enough that people keep using it. Those are engineering questions, not model questions, and they are exactly the questions a customer asks in the room right before they decide to sign or walk away. The engineer who owns cost and latency is the one who gets things shipped.

So I have learned to treat cost and latency as part of the feature, not as cleanup after it. A capability that is too expensive or too slow to deploy is not really a capability yet. It is a promise, and the job is to keep the promise.