Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Explainable LLM Risk Signals: Making Every AI Suggestion Traceable to Evidence

Build LLM risk signals auditors can trust: retrieval design, JSON schema outputs, citation checks, evaluation sets, and guardrails for ERM platforms.

techcirkle
5 min read
Cover image for Explainable LLM Risk Signals: Making Every AI Suggestion Traceable to Evidence

Putting a language model in front of incident tickets and asking "is this a risk?" takes an afternoon. Getting an internal auditor to accept the answer takes a lot more engineering. In risk management, a fluent but unsupported sentence is worse than no sentence at all, because it can end up in a board pack read by directors with legal oversight duties. This post focuses on one narrow problem: designing LLM-generated risk signals so each one can be traced, tested, and defended.

Business team analyzing risk charts and graphs on a large digital screen
Business team analyzing risk charts and graphs on a large digital screen

For the broader picture of where AI fits in enterprise risk platforms, and how to decide between buying, extending, or building one, see our full guide to ERM software solutions and AI-assisted risk sensing.

The Requirement, Stated Plainly

An AI-suggested risk signal is only usable if a reviewer can answer three questions in under a minute:

  1. What did the model look at? The exact passages, not "the incident database."

  2. What did it conclude? A specific risk ID, a direction of change, and a confidence level.

  3. Why? A short rationale that references those passages and nothing else.

Everything below is in service of those three questions.

Retrieval Design: Give the Model Less, Not More

The instinct is to stuff as much context as possible into the prompt. Resist it. For risk classification, a smaller, well-chosen context produces more checkable outputs.

For each incoming document (say, a new incident ticket):

  • Chunk the document into passages with stable IDs (doc_8812#p3).

  • Retrieve candidate risks, not just similar documents. Embed each risk's definition, scope notes, and example events, then pull the top five to eight candidates for the ticket. This bounds the label space the model chooses from.

  • Retrieve a few prior accepted examples for those candidate risks, so the model sees how reviewers have classified similar events.

  • Enforce permissions at retrieval time. The user or service account requesting classification must be entitled to every chunk returned. Risk data includes litigation and security details; leakage through a prompt is still leakage.

Structured Outputs With a JSON Schema

Free-text answers are impossible to validate. Define the output contract up front and reject anything that does not conform. Here is a TypeScript sketch using Zod for the schema and a post-validation step that checks citations against what was actually retrieved:

import { z } from "zod";

const RiskSignal = z.object({
  riskId: z.string().regex(/^[A-Z]{2,4}-\d{3}$/),
  direction: z.enum(["increase", "decrease", "no_change"]),
  confidence: z.enum(["low", "medium", "high"]),
  rationale: z.string().max(600),
  citations: z.array(z.object({
    chunkId: z.string(),
    quote: z.string().max(240),
  })).min(1),
});

const ClassificationResult = z.object({
  signals: z.array(RiskSignal).max(3),
  abstain: z.boolean(),
  abstainReason: z.string().optional(),
});

type Retrieved = { chunkId: string; text: string };

export function validateResult(raw: unknown, retrieved: Retrieved[], candidateRiskIds: Set<string>) {
  const parsed = ClassificationResult.parse(raw);
  const byId = new Map(retrieved.map(r => [r.chunkId, r.text]));

  const signals = parsed.signals.filter(s => {
    if (!candidateRiskIds.has(s.riskId)) return false;          // no invented risk IDs
    return s.citations.every(c => {
      const text = byId.get(c.chunkId);
      return text !== undefined && text.includes(c.quote);      // quote must be verbatim
    });
  });

  return { ...parsed, signals, droppedCount: parsed.signals.length - signals.length };
}

Three design choices are doing the heavy lifting:

  • Verbatim quotes. Requiring an exact substring from a retrieved chunk makes citations mechanically checkable. Paraphrased "citations" cannot be verified at scale.

  • A closed label set. The model can only return risk IDs from the retrieved candidates. Anything else is dropped, not corrected.

  • An explicit abstain path. If the model is not allowed to say "this is not a risk event," it will invent one. Abstention is a valid, logged outcome.

Most model providers now support schema-constrained or tool-call outputs, which reduces parse failures. Keep the post-validation step anyway; schema conformance does not mean the citations are real.

Prompt Design That Supports Traceability

A few prompt rules that consistently help:

  • State that conclusions must be supported only by the provided passages, and that missing evidence should lead to abstention.

  • Provide each candidate risk with its definition and explicit exclusions ("does not cover planned maintenance").

  • Ask for the rationale after the citations in the schema order, so the model commits to evidence before explaining.

  • Version prompts like code. Every logged result records the prompt version and model version.

Risk management and assessment concept graphic interface with charts and indicators
Risk management and assessment concept graphic interface with charts and indicators

Evaluation Sets: Test Before You Trust

You cannot claim explainability without measuring it. Build a labeled evaluation set from historical documents that risk owners have already classified, including plenty of negatives that should produce abstention.

Track at least these metrics per release:

Metric

What it tells you

Label precision and recall per risk category

Is the model finding the right risks?

Abstention accuracy

Does it stay quiet on non-events?

Citation validity rate

Share of signals passing verbatim quote checks

Dropped-signal rate

How often post-validation discards output

Reviewer acceptance rate (production)

Real-world usefulness, watched for rubber-stamping

Run the suite in CI on every prompt or model change, and block releases that regress beyond an agreed tolerance. Keep the eval set under version control with clear ownership, because it effectively becomes part of your control evidence.

Guardrails Around the Signal

Beyond output validation, a few platform rules keep LLM signals from turning into an audit finding:

  • Signals are proposals. They create review tasks; they never update a residual score or a control mapping directly.

  • Log the whole chain. Input document IDs, retrieved chunk IDs, prompt version, model version, raw output, validated output, and the human decision, linked by one trace ID.

  • Register the model as a risk. Give it an owner, a validation schedule, drift monitoring, and a documented fallback (deterministic rules only) if it degrades.

  • Rate-limit noise. If one document produces several signals, cap them and surface the highest-confidence few. Reviewer fatigue is how rubber-stamping starts.

If you are building this layer and want a second set of eyes on retrieval, schema design, or evaluation, our LLM integration team does this work for regulated B2B companies, and you can reach out here.

Frequently Asked Questions

Why require verbatim quotes instead of paraphrased citations?

Verbatim substrings can be checked automatically against the retrieved chunks, so fabricated or distorted evidence is caught before a reviewer sees it. Paraphrases require human judgment to verify, which does not scale across thousands of documents.

How large should an LLM evaluation set be for risk classification?

Start with a few hundred labeled examples spread across your main risk categories, with a meaningful share of negatives. Grow it continuously by adding reviewer-corrected production cases, especially ones the model got wrong.

Is JSON mode enough to make LLM outputs reliable?

No. Schema-constrained output reduces formatting errors, but it does not guarantee that risk IDs exist or that cited passages are real. Always run semantic post-validation against your candidate set and retrieved context.

How do I detect drift in an LLM risk classifier?

Compare the distribution of predicted categories, abstention rates, and reviewer acceptance rates over time, and rerun the evaluation set on a schedule. A sudden shift after a model or prompt update is the most common trigger.

Should the model see historical risk scores when classifying documents?

Usually not. Including current scores can anchor the model toward confirming existing assessments. Let it classify the evidence, and let the scoring workflow combine that signal with KRIs and owner judgment.

Subscribe to techcirkle