
A pattern we see in AI apps that come to us with margin trouble: the subscription price is sensible, conversion is decent, users love the feature. And yet gross margin keeps sliding as usage grows.
Nine times out of ten, the fix is not a new pricing page. It is in how the feature calls the model. The same capability can cost very different amounts depending on which model handles it, how much text you send, whether you have answered the same question before and whether it needs a server at all.
Every app monetization strategy for an AI product, from freemium with a credit allowance to hybrid subscription plus top-ups, assumes a cost per action. This post is about engineering that number down and keeping it visible.
Lever 1: Route requests to the smallest model that works
The prototype usually uses one large model for everything because that is the fastest way to get a demo working. In production, most requests do not need it.
function pickModel(task: Task): ModelId {
switch (task.kind) {
case 'classify':
case 'extract':
case 'short_reply':
return 'small';
case 'summarize':
return task.inputTokens < 4000 ? 'small' : 'medium';
case 'long_form':
case 'complex_reasoning':
return 'large';
}
}
Start with a static table like this, then refine it with evaluation data. A cheap pattern is escalation: try the small model, check the output against a simple validator (schema, length, confidence), and only retry with the larger model if it fails. Track the escalation rate; if it climbs, the routing rule is wrong.
Free-tier traffic is a natural place to default to smaller models, which caps exposure without switching the feature off.
Lever 2: Cache what repeats
Many AI apps answer the same request repeatedly: popular templates, common prompts, standard rewrites, identical inputs from retries.
-
Exact cache: hash the normalized prompt plus model and parameters; store the output with a TTL.
-
Semantic cache: embed the request and reuse a stored answer above a similarity threshold. Use it only for features where a near-match answer is acceptable, and never across users for anything personal.
-
Provider prompt caching: if your provider discounts repeated prompt prefixes, put the stable parts (system prompt, instructions, shared context) first and the variable user input last.
Log cache hit rate per feature. It is one of the few metrics that maps directly to money saved.
Lever 3: Trim the context
Input tokens are often the larger share of the bill, and they creep. Chat features resend the whole history. Document features send the entire file when a section would do.
function buildContext(history: Msg[], doc: Chunk[], query: string) {
const recent = history.slice(-6); // last few turns verbatim
const summary = summarizeOlder(history.slice(0, -6)); // cached rolling summary
const relevant = topK(doc, query, 5); // retrieve, don't dump
return [systemPrompt, summary, ...recent, ...relevant, query];
}
Three habits help: keep a rolling summary instead of the full transcript, retrieve relevant chunks rather than pasting whole documents, and set a hard token budget per feature that the builder enforces.

Lever 4: Move suitable work on-device
Small models running on the phone have close to zero marginal cost for you. Good candidates are tasks that are frequent, latency-sensitive and tolerant of a smaller model: autocomplete, classification, short rewrites, simple image adjustments, on-device transcription of short clips.
The trade-offs are app size, device coverage and update cadence, so treat on-device as a routing target, not a replacement:
if (device.supportsLocalModel && task.kind in localCapable) {
return runLocal(task); // near-zero marginal cost, better privacy
}
return runRemote(task, pickModel(task));
Moving even one high-frequency feature on-device can make a generous free tier affordable, with a privacy benefit attached.
Lever 5: Put a budget in the gateway
Route every model call through one internal gateway rather than calling providers from scattered services. The gateway is where you enforce:
-
per-user and per-tier daily limits,
-
per-feature token ceilings,
-
fallback to a cheaper model or a queue when a budget is hit,
-
alerts to an owner when spend deviates from forecast.
It is also the only place you can reliably attribute cost to a user and a feature.
Lever 6: Build a cost-per-user dashboard
You cannot price what you have not measured, and averages hide the problem. Under a flat subscription, light users subsidize heavy ones, which is fine until a small group consumes most of the spend.
Record cost on every call, then look at the distribution:
SELECT
width_bucket(monthly_cost, 0, 20, 10) AS cost_bucket,
count(*) AS users,
avg(monthly_revenue) AS avg_revenue
FROM (
SELECT user_id,
sum(cost_usd) AS monthly_cost,
max(plan_price) AS monthly_revenue
FROM ai_usage
WHERE ts >= date_trunc('month', now())
GROUP BY user_id
) per_user
GROUP BY cost_bucket
ORDER BY cost_bucket;
The panels worth having: cost per user by plan, the share of spend from the top slice of users, gross margin per user including inference, cache hit rate and model mix per feature. When the top bucket's cost exceeds its revenue, that is your signal to add a usage component, such as an included allowance plus top-ups.
Summary
| Lever | What it reduces | Watch metric |
|---|---|---|
| Model routing | Price per call | Escalation rate |
| Caching | Number of paid calls | Hit rate |
| Context trimming | Tokens per call | Input tokens per request |
| On-device | Remote calls entirely | Share of local executions |
| Gateway budgets | Runaway spend | Budget breaches |
| Cost dashboard | Blind spots | Margin per user |
Pricing and engineering are one decision for AI apps. For the business side, covering the seven monetization models, freemium design, store fees and pricing experiments, read the full guide on TechCirkle. If you want help putting a gateway and cost controls around your model calls, that is core work for our LLM integration team.
Frequently Asked Questions
What usually drives AI inference costs up in apps?
Using one large model for every task, resending long conversation histories or whole documents, recomputing identical answers and running calls with no per-user limits. These are wiring issues rather than pricing issues.
What is model routing?
Sending each request to the smallest model that can handle it well, and escalating to a larger model only when needed. Simple tasks such as classification or short replies rarely require the most capable model.
Is semantic caching safe to use?
It can be for features where a close match is an acceptable answer, such as common templates or generic explanations. Avoid it for personal or user-specific content, and never share cached personal outputs between users.
When does on-device AI make sense?
For frequent, latency-sensitive tasks that a small model handles well. It removes most of the marginal cost for those requests and improves privacy, at the expense of app size and device coverage.
What should a cost-per-user dashboard show?
Cost per user by plan, the distribution of spend across users, gross margin per user including inference, cache hit rates and which models each feature uses. The distribution matters more than the average.
How do these levers affect pricing decisions?
Lower and more predictable cost per action lets you offer a more generous free allowance and set subscription prices with healthier margins. Visibility into heavy users tells you when to add credits or a higher tier.