Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Shipping an LLM Compliance Pre-Check Without Letting It Approve Anything

Build an LLM pre-check for brokerage documents: narrow prompts, evidence-cited findings, per-check evals, false positive budgets and shadow mode.

techcirkle
5 min read
Cover image for Shipping an LLM Compliance Pre-Check Without Letting It Approve Anything
Agent guiding a client through signing a property contract at a desk with a house model
Agent guiding a client through signing a property contract at a desk with a house model

Brokerage compliance teams review every document in every deal: purchase agreements, addenda, disclosures, pre-approval letters. A large share of what they catch is mechanical. Missing initials. A blank mandatory field. The property address on the counter-offer not matching the one on the contract. That makes a pre-review check an obvious job for a language model inside transaction management real estate software.

It is also an easy place to cause harm. A model that approves a file with a missing disclosure creates supervision liability for the broker. So the goal is precise: a system that returns incomplete files to agents before a human ever opens them, and that is structurally incapable of approving anything.

This post covers how we would build it. For where it fits in the broader platform decision, see our guide to real estate transaction management software.

Constrain the output space first

The single most important safety decision happens before any prompt is written. The pre-check can return only three verdicts per check: pass, flag or unable_to_assess. There is no approve. Approval lives in a different service that accepts only authenticated human principals.

from pydantic import BaseModel, Field
from typing import Literal

class Evidence(BaseModel):
    page: int
    quote: str = Field(max_length=300)
    bbox: tuple[float, float, float, float] | None = None

class Finding(BaseModel):
    check_id: str                    # e.g. "buyer_initials_present"
    verdict: Literal["pass", "flag", "unable_to_assess"]
    reason: str = Field(max_length=280)
    evidence: list[Evidence]
    confidence: float

Validate every response against this schema. Anything that fails parsing becomes unable_to_assess and goes to a human.

One narrow check per call

Resist the mega-prompt that asks the model to "review this contract for compliance issues". It is impossible to evaluate and its failures are unpredictable. Instead, define a catalogue of specific checks, each with its own prompt template, version and threshold.

check_id: address_consistency
version: 4
inputs: [purchase_agreement.property_address, counter_offer.full_text]
instruction: >
  Determine whether the property address in the counter offer refers to the
  same property as the purchase agreement address provided. Ignore formatting
  differences such as abbreviations. Quote the address text you relied on.
  If no address appears, return unable_to_assess.

Put deterministic code in front wherever you can. If a known state form has a signature field at fixed coordinates, detect ink in that region with a rule. Use the model for what rules struggle with: unfamiliar addenda, handwritten counter-offers, and reasoning across documents.

Ground every finding

A flag that says "buyer initials missing" is only useful if a reviewer can verify it in seconds. Require evidence and check it programmatically:

  • Every quote must fuzzy-match text on the cited page of the OCR layer above a similarity threshold.

  • Page numbers must exist in the document.

  • A pass still needs evidence. No evidence, no pass.

Findings that fail grounding are downgraded to unable_to_assess. This one filter removes a large class of hallucinated results before anyone sees them.

Compliance reviewer comparing an AI-flagged contract page against the original document
Compliance reviewer comparing an AI-flagged contract page against the original document

Build the eval set from your own history

Brokerages have something most AI projects lack: years of labelled outcomes. Files were approved, returned or corrected, often with notes. Turn a few hundred anonymised historical files into a golden set, labelled per check, not per file.

Measure each check separately:

  • Recall on flags: of the documents that truly had the issue, how many did the check catch?

  • Precision on flags: of the flags raised, how many were real?

  • Unable-to-assess rate: how often does it punt to a human?

Different checks deserve different trade-offs. For "mandatory disclosure present", you want very high recall and can tolerate some false positives, because a missed disclosure is serious and a false flag costs an agent a minute. For "date inside expected window", precision may matter more, since noisy flags train agents to ignore the tool.

Run the eval suite in CI on every prompt, model or threshold change. A prompt edit that improves one check and quietly degrades another should fail the build.

Budget your false positives

False positives are not just an accuracy metric; they are an adoption risk. If agents receive three bogus correction requests a week, they stop reading them. Set an explicit false positive budget per check, agreed with the compliance lead, and tune thresholds against it. Checks that cannot meet their budget stay in shadow mode until they can, or get routed silently to the coordinator instead of returned to the agent.

Shadow mode before production

Run the pre-check against every live file for several weeks with results hidden from agents. Compliance staff review as usual, and the system logs where its findings agree or disagree with human decisions. Each disagreement is triaged: model wrong, human missed it, or the rule itself is ambiguous.

Only promote a check to agent-visible once its measured precision and recall on live traffic meet the agreed thresholds.

Human-in-the-loop as a data loop

When a reviewer overrides a finding, capture the reason from a short picklist plus optional text. Those overrides feed the golden set, which feeds the next eval run. Log model id and version, prompt template version, raw output, grounding result and the human decision for every document, so any approval can be reconstructed later.

Keep the model away from anything irreversible: no approvals, no client messages sent without review, and nothing that touches wiring instructions.

If you are planning a pre-check like this on top of an existing platform, our LLM integration team can help scope the checks and the evaluation harness.

Frequently Asked Questions

Why split compliance review into many narrow LLM checks?

Narrow checks are measurable. Each has its own prompt, threshold and eval metrics, so you can see exactly which one fails and fix it without destabilising the rest. A single broad prompt hides failures.

What does "unable_to_assess" add over pass and flag?

It gives the model a safe exit when the document is unreadable, the field is absent or grounding fails. Those cases go to a person instead of being forced into a misleading pass or a noisy flag.

How do you detect hallucinated findings automatically?

Require quoted evidence with page numbers and verify that each quote matches the OCR text on that page. Findings that cannot be grounded are downgraded and routed to human review.

Which metrics matter most for a compliance pre-check?

Per-check recall and precision on flags, plus the unable-to-assess rate. Critical checks like missing disclosures prioritise recall, while noisier checks may prioritise precision to protect agent trust.

How long should shadow mode run?

Long enough to see a representative mix of deal types, forms and offices, commonly several weeks. Promote each check only when its measured performance on live files meets thresholds agreed with compliance.

Where should human overrides go?

Into a structured log with a reason code, then into the golden evaluation set. That closes the loop so every prompt or model change is tested against the real mistakes reviewers have already corrected.

Subscribe to techcirkle