Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Technical Due Diligence on a Dev Shop, in One Afternoon

Sales calls tell you what an agency wants to be. A repository, a CI pipeline and a bus-factor question tell you what it actually is.

techcirkle
5 min read
Cover image for Technical Due Diligence on a Dev Shop, in One Afternoon

If you are the engineer pulled into a vendor selection, your job is not to evaluate the pitch. Someone else is doing that. Your job is to answer one question: can these people actually build and operate software?

That question is answerable in an afternoon, provided you ask for artefacts rather than opinions. Here is the sequence I use.

1. Ask for a real pull request

Not a repository. A single merged pull request from a recent project, with its review thread intact. Under NDA, with client permission, redacted as needed.

What you are reading for is not the code. It is the conversation. Specifically:

  • Did anyone push back? A review thread with three approvals and no comments means either the change was trivial or nobody read it.

  • Were the comments about substance or style? A team whose review culture is entirely about formatting has automated the wrong thing and reviews the wrong thing.

  • How did the author respond to a challenge? Defensiveness in a review thread is a reliable predictor of defensiveness in a project.

  • Was the PR small? Large diffs are a review-quality problem and a discipline signal.

A firm that cannot produce a single reviewable PR has either never worked in a shared codebase or does not want you to see how they do.

2. Ask what their pipeline does and how long it takes

Two numbers: what runs on every commit, and how long from merge to deployed.

Good answers are specific and slightly apologetic — "unit and integration on every PR, e2e nightly because the full suite takes eleven minutes and we haven't got that down yet." That is a team that has measured something and knows where it hurts.

Bad answers are vague and confident. "We have full CI/CD" tells you nothing. Follow up with: what breaks the build? If the answer is only compilation, tests are advisory, and advisory tests decay.

3. Ask for coverage — then ignore the number

Coverage as a percentage is nearly useless as a quality signal. Ask for it anyway, because the response is informative.

A team that says "about 70% on the domain layer, much lower on the UI, and here's why we made that trade" is thinking about testing strategically. A team that quotes 95% is either testing getters or lying. A team that has no idea is not measuring.

The better follow-up: what is the last bug that reached production, and what did you change so it could not happen again? You are testing for a functioning feedback loop. Teams without one repeat the same class of defect indefinitely.

4. The bus factor question

Ask directly: if the lead engineer on our project left in month three, what happens?

This is uncomfortable, which is exactly why it works. You are looking for evidence of structural resilience — documented decisions, shared context, pairing, more than one person who understands the architecture.

Watch for the firm that answers by reassuring you their people never leave. That is not an answer, it is a deflection, and staff turnover in this industry means it is also not true.

5. Ask about their AI review standard

Every agency now uses AI coding tools. The interesting question is not whether, it is what they changed about review when they adopted them.

A team that has thought about it will mention something concrete: diff-size limits, requiring the author to state intent in their own words, semantic review against requirements rather than reading the diff, mutation testing to catch assertions that cannot fail.

A team that says "we review it the same as any other code" has not noticed the specific problem — that generated code is uniformly confident and well-formatted regardless of whether it is correct, which breaks every heuristic reviewers use to decide where to look hard.

Follow-up worth asking: does our code get transmitted to third-party model providers, which ones, and are they barred from training on it? This belongs in the contract and frequently is not.

6. Ask what they ripped out

For any AI capability claim, ask what approach they tried that failed and had to be replaced.

Everyone who has shipped an AI feature to real users has this story — a retrieval strategy that fell apart at scale, a prompt chain too brittle to maintain, an agent that worked in testing and did something expensive in production. The story is specific, slightly embarrassing, and impossible to fabricate convincingly.

No story means no production experience, whatever the case studies claim.

7. Ask what they would cut

Give them your scope and ask what they would remove and why.

This does two things. It reveals whether they have actually engaged with your problem or are pattern-matching to a template. And it tells you whether they are willing to argue with a prospect — because a vendor who will not push back during the sale certainly will not push back once they are invoicing you.

The answer is also free consulting, and occasionally the most valuable output of the entire evaluation.

What the pattern tells you

You are not looking for perfect answers. You are looking for specificity. Teams that operate real software in production have concrete, slightly awkward answers full of numbers and trade-offs they are not entirely happy with. Teams that do not have smooth, general answers with no rough edges.

Smoothness is the signal. Real engineering is lumpy.

The wider guide — vendor archetypes, 2026 rate ranges, contract terms and a ten-point checklist — is here, and our take on agentic workflow development covers the AI questions in more depth.

Frequently Asked Questions

What should I ask for instead of a portfolio?

A single merged pull request with its review thread intact, under NDA. The code matters less than the conversation around it — whether anyone pushed back, whether comments addressed substance or formatting, and how the author responded to challenge. That thread reveals engineering culture in a way case studies cannot.

Is test coverage a useful quality metric?

The number is nearly useless; the response to the question is not. A team explaining why coverage is high on the domain layer and low on the UI is thinking strategically. A team quoting 95% is testing trivia. A team that does not know is not measuring — and cannot be improving.

Why does the bus factor question matter so much?

Because it tests structural resilience rather than intent. You want evidence of documented decisions, shared context and more than one person who understands the architecture. A firm that answers by promising their people never leave has deflected rather than answered, and turnover in this industry makes the promise untrue anyway.

How do I tell whether an agency's AI experience is real?

Ask what approach they tried that failed and had to be replaced. Everyone who has shipped AI features to real users has a specific, slightly embarrassing story — a retrieval strategy that broke at scale, a brittle prompt chain, an agent that misbehaved in production. It is very hard to fabricate convincingly and absent in teams that have only prototyped.

What does a good answer sound like generally?

Specific, numeric and slightly apologetic. Real engineering teams have measured things they are not entirely happy with and will tell you the number and the trade-off. Smooth, general, confident answers with no rough edges usually indicate a team describing an aspiration rather than a practice.

Subscribe to techcirkle