
Explaining a medical bill is close to an ideal use case for a language model. The inputs are structured, the vocabulary is bounded, and the desired output is two plain sentences describing a mechanical process.
It is also a use case where being wrong is expensive — financially for the patient, and in regulatory terms for the organisation. Which makes the interesting engineering problem not generation but refusal: building a system that reliably declines when it cannot support an explanation with actual data.
Most implementations treat refusal as an afterthought. It should be the primary design constraint.
What the model is actually being asked to do
The input is an adjudicated claim. Roughly:
charge_amount 850.00
allowed_amount 412.00
procedure_code [CPT]
adjustment_codes [CO-45, PR-1]
deductible_applied 312.00
coinsurance 100.00
paid_by_payer 0.00
patient_balance 412.00
The desired output is something like: "Your insurer's negotiated rate for this procedure was $412, lower than the $850 originally billed. Because you had not yet met your deductible, $312 was applied to it, and $100 is your coinsurance share."
That is a translation task over structured data, not an open-ended reasoning task. The model is not deciding anything — it is rendering a computation into language.
Framing it that way matters, because it tells you where the failure surface is. The risk is not bad reasoning. It is the model filling gaps when the structured input is incomplete.
Where hallucination actually enters
Not where people expect. The model rarely invents an amount when the amount is present. It invents reasons when the reason is absent.
Specific failure sources:
Missing or unmapped adjustment codes. There are hundreds of these. If your mapping table lacks the one on this claim, the model will produce a plausible reason rather than admit ignorance.
Partial data. The claim posted but the remittance advice has not. Everything looks structurally valid; the explanatory content is missing.
Codes with contextual meaning. Some adjustment reasons depend on plan specifics the model does not have. The code is present and mapped, but its meaning here differs.
Multi-claim situations. Two encounters, overlapping adjustments, one balance. The model will construct a coherent narrative across data that does not actually connect.
The last one is the most dangerous, because the output looks the most confident.
Design: make refusal structural, not instructed
Telling a model to say "I don't know" when uncertain is weak. It works most of the time, which is the problem — most of the time is not a compliance posture.
Better to make the explanation impossible to generate when the data does not support it.
1. Precompute the explanation skeleton in code.
Do not hand raw claim data to the model and ask for an explanation. Compute the explanation deterministically first, in ordinary application code:
explain(claim):
facts = []
if claim.allowed < claim.charged:
facts.append(NEGOTIATED_RATE, charged, allowed)
for code in claim.adjustment_codes:
mapping = ADJUSTMENT_MAP.get(code)
if mapping is None: return CANNOT_EXPLAIN(code)
facts.append(mapping, amount)
if claim.deductible_applied > 0:
facts.append(DEDUCTIBLE, amount)
...
if not covers_full_balance(facts, claim): return CANNOT_EXPLAIN(gap)
return facts
The model's job becomes rendering a validated fact list into fluent prose. If the fact list cannot be constructed, no prompt is issued at all. Refusal happens before inference.
This is the single highest-leverage decision in the design, and it converts a probabilistic safety property into a deterministic one.
2. Constrain the model to the facts provided.
With the skeleton computed, the prompt contains only validated facts and an instruction to render them without addition. Structured output helps — asking for a JSON object whose fields map to specific facts is much harder to embellish than free prose.
3. Validate the output against the input.
Post-generation, check that every number appearing in the output exists in the fact list. This is a regex-and-set-membership check, not a model call, and it catches transposition and invention cheaply.
4. Make the refusal path good.
A refusal should not be a dead end. "We can't fully explain this charge yet — here's what we know, and here's how to reach someone who can." Partial explanation of the validated portion, plus escalation, is far better than either a fabrication or a blank.
Evaluation
You need an eval set, and it must be built with billing domain experts rather than by engineers guessing.
Structure it in three parts:
Explainable cases — complete data, expected explanation content. Assert facts present, no facts absent, numbers correct.
Should-refuse cases — deliberately incomplete or unmapped. Assert refusal. This is the important half and it is the half most teams omit. An eval set containing only happy paths measures fluency, not safety.
Adversarial cases — multi-claim overlaps, unusual adjustment combinations, zero and negative balances, refunds. These are where confident wrong output appears.
Run it in CI. Run it against candidate model versions before migrating, because prompts tuned against one version degrade silently against another.
Cost, briefly
At production volume this matters. Two things help substantially:
Cache aggressively. Identical fact-list shapes produce identical explanations. Hash the skeleton and cache the rendering. In practice a large share of bills are structurally identical, so hit rates are high.
Use a small model. This is a rendering task over validated facts, not reasoning. A small model handles it well, and the deterministic skeleton means you are not relying on model capability for correctness.
Teams doing this well often find inference cost is nearly negligible, precisely because the deterministic layer did the hard part.
Why this pattern generalises
The structure here — deterministic computation, model as renderer, output validated against input, refusal before inference — applies to most regulated-domain explanation tasks. Anywhere the correct answer is computable and only the phrasing is hard, this is the shape.
It is a less exciting architecture than "give the model the data and let it reason," and it is the reason the system can be operated in production without someone reviewing every output. Serious LLM integration in regulated domains usually looks more like this and less like a chat interface.
Frequently Asked Questions
Why is bill explanation a good LLM use case? Because the inputs are structured, the vocabulary is bounded, and the output is a short natural-language rendering of a mechanical computation. The model is not reasoning about anything — it is translating a validated set of facts into fluent prose.
Where does hallucination actually occur in this task? In reasons rather than amounts. Models rarely invent a number that is present in the input, but they readily invent an explanation when an adjustment code is unmapped, when remittance data has not posted, or when multiple claims overlap and no coherent single narrative exists.
How do you make refusal reliable rather than probabilistic? Compute the explanation skeleton deterministically in application code first. If the fact list cannot be constructed — an unmapped code, a gap that does not account for the balance — no prompt is issued at all. Refusal happens before inference rather than being requested of the model.
What should an evaluation set contain? Three parts: explainable cases with expected content, should-refuse cases with deliberately incomplete data, and adversarial cases such as multi-claim overlaps and negative balances. The should-refuse half is the one most teams omit, and it is the half that measures safety rather than fluency.
Does this approach need a large model? No. Once the fact list is computed deterministically, the model is performing a rendering task rather than reasoning, and a small model handles it well. Combined with caching on fact-list shape, inference cost tends toward negligible.
What should happen when the system refuses? Explain the validated portion, state plainly that the remainder cannot yet be explained, and provide a clear escalation path to a person. A refusal that dead-ends is nearly as bad for the patient as a fabrication, even though it is far safer for the organisation.
The full guide to evaluating healthcare payment platforms is at TechCirkle.