
A model that performed well at the site where it was validated can perform meaningfully worse at yours, and the reasons are mechanical rather than mysterious. Knowing which mechanism is at work tells you what to measure.
Mechanism one: prevalence changes what a threshold means
Sensitivity and specificity are properties of the model. Positive predictive value is not — it depends on the base rate in the population being scored.
Take a model with 85% sensitivity and 90% specificity. At a 5% prevalence, roughly one in three positive alerts is a true positive. At 1% prevalence, closer to one in twelve. The model has not changed. The clinical experience of using it has changed completely, and the third alert in a shift that turns out to be nothing is what determines whether the tool survives.
This is why a tertiary referral centre and a community hospital can deploy the same model at the same threshold and reach opposite conclusions about it. Both are right about their own experience.
What to measure: your local prevalence for the target condition first, then the alert burden per clinician per shift at the vendor's default threshold. Do this before deployment, not after the complaints.
Mechanism two: documentation shifts the input distribution
Models consuming clinical text or structured fields inherit the conventions of the site that produced the training data. Template design, macro usage, specialty phrasing, and how much gets recorded as structured data versus narrative all vary between institutions and within one institution over time.
The subtle version: a site changed its templates two years ago, nobody logged it, and a model now sees a systematically different input distribution for records before and after that date. If your evaluation sample spans the change, the result is an average of two different behaviours.
What to measure: performance stratified by site and by time period. If they diverge, look for a documentation regime change before concluding anything about case mix.

Mechanism three: coding residue
Every terminology mapping has a remainder. Local codes reconciled to SNOMED, LOINC or RxNorm typically map cleanly for eighty to ninety percent, and the rest gets assigned to something adjacent.
That residual is not random. It concentrates in newer procedures, in specialty-specific concepts, and in exactly the areas where a model is most likely to be applied — because the interesting clinical questions are often the recently-codified ones.
What to measure: the unmapped and approximately-mapped fraction for the specific concepts your use case depends on, rather than for the estate. Then check whether performance differs on records where those concepts are involved.
Calibration deserves its own paragraph
Discrimination — how well the model separates cases from non-cases — often transfers reasonably. Calibration, whether a predicted probability of 0.3 corresponds to a 30% observed rate, transfers much worse, and it is the property that matters when a number is shown to a clinician who will act on it.
A model can preserve its area-under-curve entirely while its predicted probabilities become systematically too high or too low on your population. Check calibration locally, plot it, and recalibrate rather than accepting the vendor's mapping from score to probability.
The practical minimum
If you can only do a few things:
-
Establish local prevalence and compute the resulting alert burden at the proposed threshold
-
Evaluate on a few hundred locally adjudicated cases with a written rubric and a second reviewer on a subset
-
Report performance stratified by the subgroups where failure would be most consequential
-
Check and correct calibration on local data
-
Schedule the whole thing to re-run quarterly, and name who has authority to withdraw the tool
That last item is the one that gets dropped, and it is what turns monitoring into decoration. Drift in healthcare is silent — performance degrades while trust stays at its original level, because nothing is designed to notice.
Full article covering data readiness, governance, engagement models and vetting: Healthcare IT Consulting Services: What Actually Moves the Needle in 2026. Integration architecture in our LLM integration work.
Frequently Asked Questions
Can we rely on regulatory clearance instead of local validation?
No. Clearance establishes that a device met a standard against a defined population and intended use. It does not establish performance on your case mix, your documentation practice, or your coding conventions, and it says nothing about calibration on your patients.
What is the difference between discrimination and calibration?
Discrimination is how well the model ranks cases above non-cases. Calibration is whether the predicted probabilities are numerically accurate. A model can discriminate well and be badly calibrated, which matters enormously when a clinician sees the number and acts on it.
How do we recalibrate without retraining?
Fit a simple mapping — Platt scaling or isotonic regression — from the model's scores to observed outcome rates on your local sample. It requires labelled local data but not access to the model internals, which makes it feasible with a vendor product.
How large a sample is needed?
A few hundred adjudicated cases for an overall read, more for rare outcomes or for meaningful subgroup precision. Power the subgroup analysis explicitly rather than assuming the overall sample covers it.
Should evaluation be retrospective or prospective?
Retrospective for the initial read because it is cheap and fast. Prospective silent-mode running — predictions generated but not shown — is the stronger basis for a deployment decision, because it exercises the real pipeline rather than a curated extract.
Who owns this work inside a health system?
Informatics, with clinical leadership involved in defining ground truth and choosing the operating point. Left entirely to an engineering or analytics team it produces technically sound work built on clinically wrong assumptions.