Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Cost-Aware LLM Routing: Protecting Gross Margin in Conversational SaaS

Protect gross margin in conversational SaaS with small versus large model routing, answer caching, per-tenant metering, fair-use limits and tiers.

techcirkle
5 min read
Cover image for Cost-Aware LLM Routing: Protecting Gross Margin in Conversational SaaS

Traditional SaaS has a comfortable property: once the product is built, serving one more request costs close to nothing. Conversational products do not have it. Every phone call or chat consumes speech recognition, language model and speech synthesis capacity, plus telephony minutes on voice. Gross margin now depends on usage, and a chatty customer base can turn a profitable plan into a loss.

This matters most in products for small businesses, such as AI receptionists and lead follow-up tools, where monthly prices are modest and volumes vary widely between a quiet accountancy office and a busy heating contractor in winter. The pattern below treats cost as a design input from day one: route by difficulty, cache what repeats, meter everything and price in line with what you measure.

A smartphone displaying an analytics dashboard inside glowing neon rings
A smartphone displaying an analytics dashboard inside glowing neon rings

Where the money goes

Start with a per-conversation cost model. For a voice call there are four meters running:

conversation cost =
    speech recognition   (seconds of caller audio)
  + model inference      (input + output tokens, per model)
  + speech synthesis     (characters or seconds of agent audio)
  + telephony            (connected minutes)

Text channels drop the speech and telephony lines and add messaging fees. The useful insight is that model inference is the line you control most directly through engineering, while the others scale with conversation length. Shorter, more decisive conversations reduce all four.

Routing: small model by default, large model on evidence

Most turns in a local business conversation are routine: a greeting, a question about opening hours, a confirmation of a time, a yes or no. A smaller, cheaper and faster model handles these well. A minority of turns are hard: a multi-step reschedule, an ambiguous request, an unhappy customer or a quote with several conditions. Those justify a larger model.

A router decides per turn. A workable version looks like this:

function chooseModel(turn: TurnContext): ModelTier {
  if (turn.flags.complaint || turn.flags.emergency) return "large";
  if (turn.pendingToolSteps > 1) return "large";
  if (turn.intent.confidence < turn.policy.minConfidence) return "large";
  if (turn.previousAttemptFailedValidation) return "large";
  return "small";
}

A few design notes:

  • Keep the router cheap. Rules and a lightweight classifier are usually enough. A router that calls a large model to decide whether to call a large model has defeated itself.

  • Escalate on failure, not just on prediction. If the small model produces a tool call that fails schema validation, or an answer that fails a guardrail check, retry that turn on the large tier.

  • Never route on cost for safety paths. Emergencies, complaints and handoff decisions take the best model available or go straight to a person.

  • Log the decision. Record the tier, the reason and the outcome per turn so the policy can be tuned from data.

Caching what repeats

Small businesses get asked the same questions all day. Three kinds of cache are worth building.

Answer cache. Responses to common factual questions, such as hours, service area and whether a service is offered, can be stored per tenant. Key the cache on tenant, knowledge version and normalised intent, so an owner's edit to the knowledge base invalidates stale answers automatically. Never share entries across tenants.

Audio cache. Fixed phrases such as the greeting, the AI and recording disclosure and common confirmations can be synthesised once per tenant voice and replayed. That removes synthesis cost and delay for the most frequent utterances.

Prompt prefix reuse. Keep the static part of the prompt, meaning platform rules, vertical instructions and stable tenant facts, in a consistent order at the front. Where a provider supports prompt caching, a stable prefix is what makes it effective. It also keeps the per-turn variable portion small.

A business person using a smartphone surrounded by analytics and strategy icons
A business person using a smartphone surrounded by analytics and strategy icons

Metering: you cannot price what you do not measure

Emit a usage event for every billable operation, tagged with tenant and conversation:

{
  "tenantId": "t_123",
  "conversationId": "c_456",
  "channel": "voice",
  "kind": "llm",
  "modelTier": "small",
  "inputTokens": 812,
  "outputTokens": 64,
  "cacheHit": false,
  "at": "2026-11-14T10:02:11Z"
}

Aggregate these into three views. Cost per conversation tells you whether the product is efficient. Cost per tenant per month, set against that tenant's plan revenue, gives per-customer gross margin. Cost per outcome, such as per booked appointment, tells you whether outcome-based pricing is viable. Review the distribution and not only the average, because a handful of heavy tenants can hide behind a healthy mean.

Fair-use limits that do not hurt the caller

Flat pricing with variable costs is one of the classic mistakes in this market. Unlimited plans and per-minute costs do not mix. Limits are necessary, but the way they are enforced matters, because the person on the line is your customer's customer.

  • Set included usage per plan in minutes or conversations, with clear fair-use terms.

  • Alert the owner as usage approaches the allowance, before it is exceeded.

  • Prefer overage charges or an automatic tier upgrade to a hard cut-off.

  • Never drop a live call or refuse an emergency because of a quota.

  • Cap abusive patterns such as repeated spam calls at the channel layer, where they cost least.

Pricing tiers that follow cost

The most durable structure combines a base subscription with usage tiers. The base covers the fixed value of the service per location. The tiers follow minutes or conversations, which track your costs. Outcome-based pricing, such as a fee per booked appointment or qualified lead, becomes possible because the product performs work, and it is easier for an owner to compare against the cost of a part-time receptionist or of jobs lost to voicemail. Offer it only once your metering shows what an outcome costs you to produce.

Sequencing

Instrument first. Ship metering with the earliest version, even if billing is a flat pilot fee, so you know what a conversation costs before you set prices. Add caching for the obvious repeats, then introduce routing with conservative escalation rules and tune from the logs. Designing this into the orchestration layer is a large part of what we do in LLM integration projects.

For the wider picture, including which services to build, architecture, compliance and launch budgets, read our full guide to AI services for local businesses.

Frequently Asked Questions

Why does gross margin depend on usage in conversational SaaS?

Each conversation consumes speech recognition, model inference, speech synthesis and telephony capacity. Unlike traditional software, the cost of serving a customer rises with how much they use the product.

How do you decide which turns go to a larger model?

Send routine turns to a small model and escalate on evidence: low intent confidence, multi-step tasks, complaints or emergencies, and any small-model output that fails validation or guardrail checks.

What is safe to cache in a multi-tenant assistant?

Per-tenant answers to common factual questions, pre-synthesised fixed phrases and stable prompt prefixes. Key caches on tenant and knowledge version so edits invalidate them, and never share entries between tenants.

How should fair-use limits be enforced?

With included allowances, early alerts and overage tiers or upgrades. Avoid hard cut-offs that would drop a live call or block an emergency, since the caller is your customer's customer.

Which pricing model best protects margin?

A base subscription combined with usage tiers by minutes or conversations is the most durable. Outcome-based pricing can work well once metering shows what each booked appointment or qualified lead costs to deliver.

Subscribe to techcirkle