Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Your AI Feature Is Priced Right. Your Inference Bill Isn't.

Model routing, caching, context trimming, on-device models and cost-per-user dashboards: the levers behind app monetization strategies that hold up.

techcirkle
5 min read
Cover image for Your AI Feature Is Priced Right. Your Inference Bill Isn't.
Person completing an in-app payment on a smartphone with a credit card, representing the revenue side of an AI app's margins
Person completing an in-app payment on a smartphone with a credit card, representing the revenue side of an AI app's margins

A pattern we see in AI apps that come to us with margin trouble: the subscription price is sensible, conversion is decent, users love the feature. And yet gross margin keeps sliding as usage grows.

Nine times out of ten, the fix is not a new pricing page. It is in how the feature calls the model. The same capability can cost very different amounts depending on which model handles it, how much text you send, whether you have answered the same question before and whether it needs a server at all.

Every app monetization strategy for an AI product, from freemium with a credit allowance to hybrid subscription plus top-ups, assumes a cost per action. This post is about engineering that number down and keeping it visible.

Lever 1: Route requests to the smallest model that works

The prototype usually uses one large model for everything because that is the fastest way to get a demo working. In production, most requests do not need it.

function pickModel(task: Task): ModelId {
  switch (task.kind) {
    case 'classify':
    case 'extract':
    case 'short_reply':
      return 'small';
    case 'summarize':
      return task.inputTokens < 4000 ? 'small' : 'medium';
    case 'long_form':
    case 'complex_reasoning':
      return 'large';
  }
}

Start with a static table like this, then refine it with evaluation data. A cheap pattern is escalation: try the small model, check the output against a simple validator (schema, length, confidence), and only retry with the larger model if it fails. Track the escalation rate; if it climbs, the routing rule is wrong.

Free-tier traffic is a natural place to default to smaller models, which caps exposure without switching the feature off.

Lever 2: Cache what repeats

Many AI apps answer the same request repeatedly: popular templates, common prompts, standard rewrites, identical inputs from retries.

  • Exact cache: hash the normalized prompt plus model and parameters; store the output with a TTL.

  • Semantic cache: embed the request and reuse a stored answer above a similarity threshold. Use it only for features where a near-match answer is acceptable, and never across users for anything personal.

  • Provider prompt caching: if your provider discounts repeated prompt prefixes, put the stable parts (system prompt, instructions, shared context) first and the variable user input last.

Log cache hit rate per feature. It is one of the few metrics that maps directly to money saved.

Lever 3: Trim the context

Input tokens are often the larger share of the bill, and they creep. Chat features resend the whole history. Document features send the entire file when a section would do.

function buildContext(history: Msg[], doc: Chunk[], query: string) {
  const recent = history.slice(-6);                 // last few turns verbatim
  const summary = summarizeOlder(history.slice(0, -6)); // cached rolling summary
  const relevant = topK(doc, query, 5);             // retrieve, don't dump
  return [systemPrompt, summary, ...recent, ...relevant, query];
}

Three habits help: keep a rolling summary instead of the full transcript, retrieve relevant chunks rather than pasting whole documents, and set a hard token budget per feature that the builder enforces.

Smartphone surrounded by business finance and operations app icons, representing per-user cost tracking across AI features
Smartphone surrounded by business finance and operations app icons, representing per-user cost tracking across AI features

Lever 4: Move suitable work on-device

Small models running on the phone have close to zero marginal cost for you. Good candidates are tasks that are frequent, latency-sensitive and tolerant of a smaller model: autocomplete, classification, short rewrites, simple image adjustments, on-device transcription of short clips.

The trade-offs are app size, device coverage and update cadence, so treat on-device as a routing target, not a replacement:

if (device.supportsLocalModel && task.kind in localCapable) {
  return runLocal(task);   // near-zero marginal cost, better privacy
}
return runRemote(task, pickModel(task));

Moving even one high-frequency feature on-device can make a generous free tier affordable, with a privacy benefit attached.

Lever 5: Put a budget in the gateway

Route every model call through one internal gateway rather than calling providers from scattered services. The gateway is where you enforce:

  • per-user and per-tier daily limits,

  • per-feature token ceilings,

  • fallback to a cheaper model or a queue when a budget is hit,

  • alerts to an owner when spend deviates from forecast.

It is also the only place you can reliably attribute cost to a user and a feature.

Lever 6: Build a cost-per-user dashboard

You cannot price what you have not measured, and averages hide the problem. Under a flat subscription, light users subsidize heavy ones, which is fine until a small group consumes most of the spend.

Record cost on every call, then look at the distribution:

SELECT
  width_bucket(monthly_cost, 0, 20, 10) AS cost_bucket,
  count(*)                              AS users,
  avg(monthly_revenue)                  AS avg_revenue
FROM (
  SELECT user_id,
         sum(cost_usd)    AS monthly_cost,
         max(plan_price)  AS monthly_revenue
  FROM ai_usage
  WHERE ts >= date_trunc('month', now())
  GROUP BY user_id
) per_user
GROUP BY cost_bucket
ORDER BY cost_bucket;

The panels worth having: cost per user by plan, the share of spend from the top slice of users, gross margin per user including inference, cache hit rate and model mix per feature. When the top bucket's cost exceeds its revenue, that is your signal to add a usage component, such as an included allowance plus top-ups.

Summary

Lever

What it reduces

Watch metric

Model routing

Price per call

Escalation rate

Caching

Number of paid calls

Hit rate

Context trimming

Tokens per call

Input tokens per request

On-device

Remote calls entirely

Share of local executions

Gateway budgets

Runaway spend

Budget breaches

Cost dashboard

Blind spots

Margin per user

Pricing and engineering are one decision for AI apps. For the business side, covering the seven monetization models, freemium design, store fees and pricing experiments, read the full guide on TechCirkle. If you want help putting a gateway and cost controls around your model calls, that is core work for our LLM integration team.

Frequently Asked Questions

What usually drives AI inference costs up in apps?

Using one large model for every task, resending long conversation histories or whole documents, recomputing identical answers and running calls with no per-user limits. These are wiring issues rather than pricing issues.

What is model routing?

Sending each request to the smallest model that can handle it well, and escalating to a larger model only when needed. Simple tasks such as classification or short replies rarely require the most capable model.

Is semantic caching safe to use?

It can be for features where a close match is an acceptable answer, such as common templates or generic explanations. Avoid it for personal or user-specific content, and never share cached personal outputs between users.

When does on-device AI make sense?

For frequent, latency-sensitive tasks that a small model handles well. It removes most of the marginal cost for those requests and improves privacy, at the expense of app size and device coverage.

What should a cost-per-user dashboard show?

Cost per user by plan, the distribution of spend across users, gross margin per user including inference, cache hit rates and which models each feature uses. The distribution matters more than the average.

How do these levers affect pricing decisions?

Lower and more predictable cost per action lets you offer a more generous free allowance and set subscription prices with healthier margins. Visibility into heavy users tells you when to add credits or a higher tier.

Subscribe to techcirkle