Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

The Tech Lead's Playbook for Building Your Own AI Feature

A practical playbook for tech leads building their own AI: rung selection, eval-first delivery, guardrails and a launch checklist plus a code example.

techcirkle
5 min read
Cover image for The Tech Lead's Playbook for Building Your Own AI Feature
A hand holding a tablet with a glowing outline of a human head, gears and speech bubbles above it on a teal background
A hand holding a tablet with a glowing outline of a human head, gears and speech bubbles above it on a teal background

You have been handed a ticket that says, more or less, "build our own AI". As the tech lead, your job is to turn that into something shippable, measurable and cheap to run. This playbook is the sequence I would follow, with a checklist at each stage and one small code example where it earns its place.

It is a condensed, practitioner version of our full guide on building your own AI, which goes deeper on cost ranges and roadmap phases.

Play 1: Reframe the request

Before any code, get the stakeholders to agree that "our own AI" means owning the layers that differentiate you (data, logic, evaluations, integrations) and renting the base model. Training from scratch is off the table unless someone can name a very specific reason.

Checklist

  • Use case written as a task with an input, an output and a person who does it today

  • A measurable success metric (time saved, error rate, deflection rate)

  • A named domain expert who can say what a correct answer looks like

  • Agreement that the first release targets the lowest viable rung

Play 2: Pick the lowest rung that could work

Run through the ladder in order and stop at the first rung that fits:

  1. Prompt plus hosted API. Enough for extraction, classification, summarising and drafting when no private knowledge is needed.

  2. RAG. Needed when answers depend on your documents or records.

  3. Fine-tuning or a small open-weight model. For consistent format or style at volume, or when data cannot leave your environment.

  4. Agents and workflows. When the task needs actions across systems, not just an answer.

  5. Pretraining. Not for you.

A quick sanity rule: facts that change go in retrieval, behaviour that must be consistent may go in fine-tuning.

Checklist

  • Rung chosen and the reason written in the design doc

  • The trigger for climbing to the next rung defined in measurable terms

Play 3: Build the eval set before the feature

This is the play most teams skip. Collect 50 to 300 real examples with expected outputs, ideally written or reviewed by the domain expert. Wire them into CI so prompt and model changes are gated on a pass rate.

Here is a minimal Python harness. It is deliberately simple; the point is to get a number on day one.

# evals/run_evals.py
import json, sys
from app.pipeline import answer  # your feature entry point

THRESHOLD = 0.85

def score(case, output):
    text = output.lower()
    hits = all(k.lower() in text for k in case["must_include"])
    clean = not any(k.lower() in text for k in case.get("must_not_include", []))
    return hits and clean

cases = [json.loads(l) for l in open("evals/cases.jsonl")]
results = [(c["id"], score(c, answer(c["input"]))) for c in cases]
rate = sum(ok for _, ok in results) / len(results)

for cid, ok in results:
    if not ok:
        print(f"FAIL {cid}")
print(f"pass rate {rate:.0%} (threshold {THRESHOLD:.0%})")
sys.exit(0 if rate >= THRESHOLD else 1)

Later you will add model-graded scoring and, for RAG, a retrieval-recall check. Keep the cases file in version control and add to it every time a user flags a bad answer.

Checklist

  • Eval set exists before the first prompt is tuned

  • CI fails on regressions

  • Separate metrics for retrieval quality and answer quality (if using RAG)

Play 4: Build the thin production spine

You do not need a platform team, but you do need these from the first production release:

  • A model abstraction layer, so a provider or version change is configuration, not a rewrite

  • Tracing of every call: inputs (masked), retrieved context, output, latency, cost

  • Per-feature and per-customer cost tracking with alerts and hard caps

  • Access control enforced at retrieval, inherited from source systems

  • A written data policy on what may be sent to which model, signed off by security and legal

  • A feedback button and a human review path for high-stakes outputs

If you are integrating models into an existing product or CRM and want a second pair of hands on the plumbing, this is the core of what our LLM integration work covers.

Play 5: Guardrails by risk

Risk

Guardrail

Hallucination

Ground in retrieved sources, require citations, constrain output schema, verify checkable facts

Data leakage

Permission filters at retrieval, treat all inputs as untrusted, scope agent tools

Prompt injection

Separate instructions from data, validate outputs before side effects

Cost runaway

Budgets per feature and tenant, step caps on agents, caching, small-model routing

Model deprecation

Abstraction layer plus eval replay before switching

Low adoption

Ship inside existing tools, keep a human in control, train users on limits

Play 6: Ship a pilot, then decide

Release to a limited group of real users. Measure task completion, time saved, error rates and satisfaction against the pre-AI baseline. Then make an explicit go, iterate or stop call.

Only after that consider climbing a rung. Fine-tuning and self-hosting should be justified by eval results and cost data, not by the feeling that they are more serious.

Play 7: Protect the assets

Make sure the company owns the code, prompts, eval datasets, pipelines and any tuned weights, including when an external team builds the first version. Check model vendor terms on training with your inputs and prefer agreements that exclude it. Once the pipelines, gateway and eval tooling exist, reuse them; the second use case should cost noticeably less than the first.

Launch checklist

  • Eval pass rate above threshold on the latest model version

  • Spend caps and alerts configured

  • Traces visible and sensitive fields masked

  • ACL filtering tested with a low-privilege user

  • Human review path live for high-stakes outputs

  • Rollback plan documented (previous prompt and model version)

  • Domain expert has signed off on a sample of outputs

Frequently Asked Questions

What is the first thing a tech lead should do with an "AI" ticket?

Rewrite it as a concrete task with inputs, outputs, a success metric and a named domain expert. Without those, there is nothing to build or measure.

How do I decide between RAG and fine-tuning?

If the knowledge changes, use retrieval so updates are just re-indexing. Consider fine-tuning only for consistent behaviour or format at high volume, after evals show prompts are not enough.

Do I need a dedicated ML engineer?

Not for prompt-based or retrieval-based features, which strong backend and data engineers can deliver. Bring in ML experience for fine-tuning, self-hosting or classical models.

What pass rate should my eval gate use?

There is no universal number. Set it from the baseline you measure on day one and the cost of an error in your domain, then raise it as the system improves.

How do I keep inference costs predictable?

Attribute cost per feature and per customer, enforce hard caps in the gateway, cache common answers and route simple requests to cheaper models once evals confirm parity.

What should the rollback plan include?

The previous prompt version, the previous model configuration and a way to switch back quickly, plus the eval results that justified the change.

Subscribe to techcirkle