We parse construction and real-estate documents with an agentic parser: invoices, payment certificates, inspection reports. The parser's output is markdown, and everything downstream (field extraction, totals, the record a person signs off) treats that markdown as the document.
In a 35-page evaluation of construction-payment documents, that assumption broke in ways that were hard to see. The first table of a tax invoice was a 100-row fabrication, complete with a plausible subtotal. An amount in Arabic-Indic digits gained a digit: ٢٢٬٢٠٠٫٠٠ became ٢٢٦٢٠٠,٠٠. Lines went missing on plans and on an update. None of it looked wrong in the markdown. Downstream extraction would have read all of it as fact.
The obvious fix is a second model reviewing the first. We did not want that. A check that is itself an LLM can be wrong in the same plausible way, and then you are trusting output you cannot audit twice instead of once. The check has to read the PDF.
dpp-check is a conformance checker for the Document Parsing Protocol (DPP), a working draft published at parseprotocol.com by LlamaIndex's run-llama project. DPP writes down the choices every parser makes on a page: what to preserve, how to encode structure, and how to make uncertainty explicit. dpp-check takes one parser's markdown for one document and checks it against the PDF itself. It is deterministic, makes no network calls, needs no API key and uses no LLM judge. It is not affiliated with or endorsed by the protocol project or LlamaIndex.
It is open source under the MIT licence and written in Rust. It ships as a CLI, a GitHub Action, and a WebAssembly build for Node and Bun, tested on Node 24.20, Node 22.23.4 and Bun 1.4.4. The current release is 0.2.1.
The rules target the failures that cost money when they slip through. Invented tables, including key/value content set out as a table, which DPP covers under forms and key-value regions (§10.1). Dropped form content, meaning blanks and checkboxes. Values that must be copied verbatim and stay bound to their own label, without truncation. Table content: rows or cells that are not on the page, changed or swapped cells, duplicated or out-of-order rows, dropped columns, and rows that continue onto a following page. Number formats, including Arabic-Indic digits and the different digit-grouping styles.
Some pages cannot be read reliably, for example glyphs that do not decode, or Arabic text without ActualText. Those are judged from the page side. A finding that cannot be verified becomes a "not verified" warning. It is never a silent pass.
A checker tuned against its own test set will look good on that set. So the numbers below are labelled by where they come from, and the ones that count most come first: data the fixes were never tuned on.
PDFs from independent producers. Seven PDFs made with typst, Ghostscript and fpdf2, built by the reviewer rather than the developer. With correct markdown, dpp-check reported 0 false errors. On the typst and Ghostscript invoices, we then planted errors in the markdown:
Independent PDFs (not tuned on) | Result |
|---|---|
False errors on correct markdown, 7 PDFs | 0 |
Planted errors caught, typst and Ghostscript invoices | 16 of 17 |
The one miss was a stripped thousands separator, which is reported as a warning by design rather than an error.
A blind set inside our own product. There, dpp-check runs as a second reader in the parse step, in a worker thread, with the module pinned by checksum. On an unseen blind set of 13 documents:
Blind set, 13 documents (not tuned on) | Result |
|---|---|
Extra correct catches over our own checks alone | 1 |
False alarms | 1 of 13 (7.7 %) |
The extra catch was a date changed from 25/10/2026 to 25/01/2026 on a bilingual certificate. That is the whole result, and we are not claiming more from it.
The first version was not useful. These two tables come from the data we developed the fixes against. They show that the fixes worked on what they were built for. They are not a measure of general accuracy.
The same 35 pages of parser output, including one reference page the parser had got exactly right:
Same 35-page corpus | v0.1.0 | v0.2.x |
|---|---|---|
Errors reported | 224 | 8 |
Of those, correct | 0 | 7 |
Errors on the perfectly parsed reference page | 125 | 0 |
A mutation harness that plants defects in correct markdown and counts how many get caught, same seed for both versions:
Planted defect | v0.1.0 caught | v0.2.x caught |
|---|---|---|
Table cell number changed | 0/40 | 40/40 |
Duplicated row | 12/40 | 40/40 |
Fabricated row | 12/40 | 40/40 |
Checkbox changes on Chromium-produced PDFs | 0/12 | 12/12 |
Arabic digit corruption | 0/8 | 4/8 |
That last row is half, and we report it as half.
It is quick enough to sit inline in a pipeline:
Speed and size | Measured |
|---|---|
Check time per page | about 2 to 3 ms |
WebAssembly module | 1.6 MB (0.56 MB gzipped) |
Open a 585-page, 2.4 MB PDF | about 41 ms |
Open a 104 MB PDF | about 1.3 s |
WebAssembly memory | about 4.3 times the file size |
The API loads a PDF once and checks against it as often as you need: open(pdf), then check(markdown, pages), then free().
It reads born-digital PDFs only. Scanned pages are reported as unevidenced rather than guessed. Arabic from producers other than Chromium mostly comes back as "not verified". Dropped prose that carries no numbers is not caught. Table header rows are treated as optional on the page. Flipped ☑/☐ glyph checkboxes in markdown are not recognised yet. On right-to-left pages, a reversed hyphenated identifier can raise a false flag.
It also does less than its name might suggest. It checks one parser's output for one document; it is not a benchmark and does not rank parsers. A clean report means it found nothing it could hold against the page, not that the markdown is complete. The "not verified" warnings are the honest boundary of what it knows, and they are where a person should look.
The source, the GitHub Action and the WebAssembly build are here, hosted on ngit (git over Nostr):
gitworkshop.dev/c@mailstr.app/relay.ngit.dev/dpp-check
Point the CLI at a PDF and the markdown your parser produced for it:
dpp-check --pdf <file.pdf> --markdown <file.md> --format json
If it flags something you think is right, or misses something it should have caught, send us the page. A failing example is the most useful thing you can give a checker.
I write about document parsing, verification and honest measurement as metaend. If you want to reach me, build on any of this, or tell me where it breaks, the door is here:
More writing lives at paragraph.com/@metaend.
Written by metaend.

