Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Keeping LLM Features Healthy in Production Mobile Apps

Maintain AI features in mobile apps: pin model versions, run prompt eval suites in CI, watch token cost drift, and plan fallbacks for deprecations.

techcirkle
5 min read
Cover image for Keeping LLM Features Healthy in Production Mobile Apps

Traditional app code only changes when someone on your team changes it. An AI feature does not follow that rule. A vendor can retire the model behind your chat assistant, users can start asking questions your prompt was never tuned for, and your token bill can creep upward while every dashboard looks green. If your mobile app now summarizes documents, answers support questions, or ranks search results with an LLM, you have inherited a new kind of maintenance. This post covers the engineering practices that keep it predictable.

Person holding a smartphone that displays an app update and repair icon
Person holding a smartphone that displays an app update and repair icon

Rule one: never call a floating model alias

Many LLM APIs offer aliases that silently point to the latest model in a family. They are convenient in a prototype and dangerous in production, because output format, tone, latency, and price can shift without a deploy. Pin an explicit, dated model version, and keep model and prompt choices on the server, not in the app binary. A mobile client should call your backend, which owns the model configuration. That way a model migration is a server change behind a flag, not an app store release that takes days to reach users.

A config shape we like:

// ai-config.ts
export const AI_FEATURES = {
  summarize: {
    primary: { provider: "vendorA", model: "model-x-2025-06-01", maxTokens: 600 },
    fallback: { provider: "vendorB", model: "model-y-2025-03-15", maxTokens: 600 },
    promptVersion: "summarize@v7",
    timeoutMs: 8000,
    monthlyBudgetUsd: 1500,
  },
} as const;

Every field is reviewable in a pull request, and changing any of them triggers the eval suite described below.

Rule two: prompts are code, so test them like code

A prompt tweak that improves one case can quietly break another. Store prompts in source control with version identifiers, and gate changes with a regression eval suite: a fixed set of representative inputs plus checks on the qualities you care about.

// evals/summarize.eval.ts
import { runFeature } from "../src/ai";
import cases from "./summarize.cases.json";

type Check = (out: string) => boolean;

const checks: Record<string, Check> = {
  underLimit: (o) => o.split(/\s+/).length <= 120,
  noMarkdown: (o) => !/[#*`]/.test(o),
  validJson: (o) => { try { JSON.parse(o); return true; } catch { return false; } },
};

let failures = 0;
for (const c of cases) {
  const out = await runFeature("summarize", c.input);
  for (const name of c.checks) {
    if (!checks[name](out)) {
      failures++;
      console.error(`FAIL ${c.id} :: ${name}`);
    }
  }
}
const passRate = 1 - failures / cases.reduce((n, c) => n + c.checks.length, 0);
console.log(`pass rate ${(passRate * 100).toFixed(1)}%`);
if (passRate < 0.95) process.exit(1);

Run it in CI on any change to prompts, model IDs, or retrieval logic. Deterministic checks (length, format, required fields, banned phrases) catch most regressions cheaply; add a model-graded rubric for fuzzier qualities like faithfulness, and spot-check those grades by hand. Record the pass rate per run so you can see drift, not just failures.

Rule three: watch cost drift as a first-class metric

Token usage grows as adoption grows, but it also grows for less obvious reasons: prompts accumulate instructions, retrieval stuffs in more context, and conversation history is never trimmed. Track:

  • Cost per active user per feature, reviewed monthly.

  • Input and output tokens at the 50th and 95th percentile, so one runaway prompt path shows up.

  • Cache hit rate if you use prompt caching or response caching for repeated queries.

  • Routing mix, if simple requests go to a cheaper model and hard ones to a larger model.

Set a budget per feature in config (as above) and alert at 80 percent of it. A budget alert caught in week two is a tuning task; one caught on the invoice is an incident.

Hand touching a tablet surrounded by virtual technical support and maintenance icons
Hand touching a tablet surrounded by virtual technical support and maintenance icons

Rule four: plan for deprecation before the email arrives

LLM vendors retire model versions on published schedules, sometimes with only months of notice. Treat each migration like an SDK upgrade:

  1. Keep a tested fallback model from a different family or vendor, configured and passing evals, so a deprecation or outage is a config flip.

  2. Wrap the provider call in a thin adapter so request and response formats are normalized in one place.

  3. Run the eval suite against the candidate model, compare pass rate, p95 latency, and cost per call, then roll out behind a server-side flag to a small percentage of users.

  4. Degrade gracefully. If both models fail or time out, the app should show a useful non-AI fallback (for example, the raw document or a search result list), never a spinner that never ends.

Rule five: on-device models need release engineering too

Apps that run models locally through Core ML, ML Kit, or similar frameworks add a different set of chores:

  • Ship models separately from the binary where possible, with versioned downloads, checksums, and resume support, so a model update does not require a store release.

  • Gate by device capability. Older devices may lack memory or neural hardware for the newest model; keep a smaller model or a server path for them.

  • Watch app size and download cost. Large model files hurt install conversion and can be expensive on metered connections; download on Wi-Fi by default.

  • Run the same evals on-device using a test harness in CI, because quantized models can behave differently from their server counterparts.

How this fits a broader maintenance plan

AI upkeep sits on top of the usual mobile work: OS upgrades, dependency patching, crash triage, and store compliance. Teams with meaningful AI features should plan for extra effort in evaluation, migration testing, and cost reviews. For the full scope of ongoing upkeep, including SLAs and pricing models, read our guide to mobile app maintenance services and how to scope them. If you are designing or hardening AI features, our LLM integration team builds with this lifecycle in mind from the start.

Frequently Asked Questions

Should a mobile app call an LLM API directly?

Usually not. Routing calls through your backend keeps API keys off devices, lets you change models and prompts without an app release, and gives you one place for logging, rate limiting, and cost controls.

How many test cases does a prompt eval suite need?

Start with 30 to 100 representative cases covering common inputs, edge cases, and known past failures. Add a case every time a regression reaches users, so the suite grows from real incidents.

How do I detect quality drift when the model version is pinned?

Input drift still happens as users change behavior. Sample production inputs (with consent and redaction), run them through your evals periodically, and watch the pass rate trend.

What is a reasonable fallback when an LLM provider is down?

A second pre-tested model behind the same adapter, and below that a non-AI path such as showing source content or standard search results. Users tolerate reduced features far better than broken screens.

How often should on-device models be updated?

Only when evals show a meaningful gain or a bug fix. Each update costs download bandwidth and testing across device tiers, so batch improvements rather than shipping every minor retrain.

Subscribe to techcirkle