Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Designing LLM Features in a CRM So the Invoice Never Surprises You

Model tiering, change-only processing, caching and acceptance-based evals: engineering patterns that keep AI CRM features cheap, accurate and honest.

techcirkle
5 min read
Cover image for Designing LLM Features in a CRM So the Invoice Never Surprises You
CRM dashboard with analytics and customer icons representing the cost and performance of AI features in a CRM
CRM dashboard with analytics and customer icons representing the cost and performance of AI features in a CRM

A familiar story: an AI feature ships in a CRM, users like it, and in month three someone in finance asks why the model provider invoice looks like a new hire's salary.

Usually the feature is fine. The wiring is not. Somebody scheduled a nightly job that runs a large model over every opportunity, whether or not anything changed. Or every summary request re-reads the full email history. Or drafting and classification share the same expensive model because that was the one in the prototype.

When you add artificial intelligence to a CRM, cost discipline is an architecture decision, not a finance review after launch. So is measuring whether each feature deserves to exist. This post covers both.

Pattern 1: Tier your models by task

Most CRM AI work falls into two buckets.

  • Extraction and classification: pull next steps from a transcript, detect an objection type, tag an email's intent, spot a champion going quiet. Structured output, short context, high volume.

  • Generation and reasoning: draft a renewal email, summarise a complicated handoff, answer a manager's open question. Lower volume, higher quality bar.

Route the first bucket to small, fast models and reserve larger ones for the second. Do it in a gateway so no feature team can quietly bypass it.

const ROUTES: Record<Feature, ModelRoute> = {
  extract_activity:  { tier: "small", maxOut: 600 },
  classify_intent:   { tier: "small", maxOut: 50 },
  deal_risk_flags:   { tier: "small", maxOut: 400 },
  draft_followup:    { tier: "large", maxOut: 1200 },
  handoff_summary:   { tier: "large", maxOut: 1500 },
};

Pattern 2: Process only what changed

The most expensive line in a CRM AI budget is usually repeated work on unchanged data. Key every derived artefact on a content hash of its inputs.

async function refreshRiskFlags(oppId: string) {
  const inputs = await loadRiskInputs(oppId);   // activity, stage history, contacts
  const key = hash(inputs, PROMPT_VERSION, MODEL_ROUTE.deal_risk_flags);

  if (await cache.has(key)) return cache.get(key);

  const result = await gateway.run("deal_risk_flags", inputs);
  await cache.set(key, result, { ttlDays: 14 });
  return result;
}

Trigger it from change events (new email, logged call, stage update) rather than a full nightly sweep. Include the prompt version and model in the key so an intentional change invalidates the cache and an accidental one does not go unnoticed.

Pattern 3: Summarise incrementally

Do not re-read a 200-message thread for every summary. Maintain a rolling summary per record and fold in only the new messages.

newSummary = gateway.run("handoff_summary", {
  previousSummary: record.summary,
  newMessages: messagesSince(record.summaryUpdatedAt),
});

Periodically rebuild from scratch to stop drift, but make that the exception.

Pattern 4: Budgets with hard ceilings

Set per-feature and per-tenant daily budgets in the gateway. When a ceiling is hit, degrade gracefully — queue non-urgent work, fall back to a smaller model, or pause the feature — and alert a human. A runaway loop should cost you a day's budget, not a month's.

Engineering and RevOps team reviewing AI feature metrics and CRM usage on a shared dashboard
Engineering and RevOps team reviewing AI feature metrics and CRM usage on a shared dashboard

Pattern 5: Measure acceptance, not volume

Cost only makes sense next to value. The cheapest honest measure of value in a CRM is what users do with each suggestion.

type Feedback = {
  suggestionId: string;
  feature: Feature;
  outcome: "accepted" | "edited" | "rejected";
  editDistance?: number;
  promptVersion: string;
  model: string;
};

From that you can compute acceptance rate per feature, cost per accepted suggestion, and trend lines by prompt version. A feature with high volume, low acceptance and rising cost is a candidate for redesign or removal. "Suggestions generated" on its own is a vanity metric.

Pattern 6: Turn rejections into your eval set

Every rejected or heavily edited suggestion is a labelled example of the system getting it wrong. Collect them, with the retrieved context that produced them, into a regression set per feature.

async function evaluateCandidate(feature: Feature, candidate: PromptConfig) {
  const cases = await evalSet.load(feature);        // real rejections + a sample of acceptances
  const results = await Promise.all(cases.map(c => runWith(candidate, c.inputs)));
  return score(results, cases);                      // rubric or reviewer-in-the-loop
}

Run it before any prompt or model change ships. Include a sample of previously accepted cases too, so you notice when a fix for one problem quietly breaks something that already worked. Budget human review time for this explicitly; someone has to own it.

Pattern 7: Tie it back to business metrics

Engineering metrics tell you whether the system is healthy. They do not tell you whether it is worth running. Pair them with a small set of outcome metrics agreed with the revenue team before launch:

  • share of opportunities with recent logged activity

  • rep hours returned from data entry, sampled rather than self-reported

  • forecast variance by quarter

  • median speed to first human touch on inbound leads

If none of those move after a reasonable period, the feature is decoration, no matter how elegant the caching layer is.

Summary

Lever

What it prevents

Model tiering

Paying large-model prices for small-model tasks

Change-only processing

Re-running work on unchanged records

Incremental summaries

Re-reading full histories every time

Gateway budgets

Runaway loops and surprise invoices

Acceptance tracking

Mistaking usage for value

Rejection-based evals

Shipping regressions with prompt changes

This post is the engineering slice of a broader guide covering use cases, the five-layer architecture, security and a build-versus-buy framework. Read the full guide on TechCirkle. If you want a hand designing the gateway and eval loop, our AI development services team does this regularly.

Frequently Asked Questions

Why do AI CRM features become expensive so quickly?

Usually because of wiring rather than usage: large models used for simple tasks, nightly jobs that reprocess unchanged records, and summaries that re-read entire histories on every request.

Which CRM tasks suit smaller language models?

Extraction and classification tasks such as pulling next steps from transcripts, tagging email intent and detecting deal-risk patterns. They need structured output and short context rather than long-form reasoning.

How should caching work for CRM AI outputs?

Key each output on a hash of its inputs plus the prompt version and model. Recompute only when something relevant changes, triggered by CRM events rather than scheduled full sweeps.

What is the best metric for an AI feature in a CRM?

Acceptance rate by feature — the share of suggestions accepted or lightly edited — ideally shown alongside cost per accepted suggestion. Raw suggestion counts say little about value.

How do I build an evaluation set for CRM AI?

Collect rejected and heavily edited suggestions with the context that produced them, add a sample of accepted ones, and replay the set against any prompt or model change before shipping.

What happens when a gateway budget is exceeded?

The gateway should degrade gracefully by queuing non-urgent work, falling back to a cheaper model or pausing the feature, and should alert an owner so the cause can be investigated.

Subscribe to techcirkle