Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Building an LLM-Ready Document Pipeline That Respects Permissions

A technical walkthrough of an unstructured data pipeline for RAG: parsing, chunking, embedding versioning, ACL-aware retrieval and evaluation sets.

techcirkle
4 min read
Cover image for Building an LLM-Ready Document Pipeline That Respects Permissions
Engineer working on a laptop displaying big data visualisations
Engineer working on a laptop displaying big data visualisations

Most retrieval-augmented generation demos index a folder once and call it done. Production is different. Documents change, get deleted, move between teams and carry access rules. The model never sees any of that unless the pipeline puts it there. This is the reference design we use when an AI feature has to answer from real company content.

Stages

sources (drive, wiki, ticketing, CRM attachments)
  -> change detection (webhooks or polling with etags)
  -> parse (text, tables, OCR) + metadata
  -> chunk (structure-aware)
  -> embed (versioned model)
  -> index (vectors + metadata + ACLs)
  -> retrieve (hybrid search + permission filter + rerank)
  -> evaluate (versioned question set)

Treat this like any other data pipeline: orchestrated, idempotent, observable, with lineage back to the source document.

Change Detection and Deletes

Index freshness is where most systems quietly fail. For every document store a stable source_id, a content_hash and a source_version. On each sync:

for doc in source.list_changed(since=cursor):
    if doc.deleted:
        index.delete(filter={"source_id": doc.id})
    elif doc.hash != registry.get_hash(doc.id):
        chunks = chunk(parse(doc))
        index.delete(filter={"source_id": doc.id})
        index.upsert(embed(chunks, model=EMBED_MODEL_V))
        registry.set(doc.id, doc.hash, EMBED_MODEL_V)

Delete-then-upsert per document avoids stale chunks coexisting with new ones, which is the classic cause of an assistant quoting last year's policy.

Parsing and Chunking

Parsing quality caps everything downstream. Preserve headings, keep tables as tables (Markdown or structured cells), and run OCR only where text extraction fails. Chunk along document structure, sections and paragraphs, with modest overlap, rather than fixed character windows. Attach the heading path to every chunk; it improves retrieval and gives the model context for free.

Version the Embedding Model

Store the embedding model identifier on every vector. When you upgrade, build a new index alongside the old one, backfill, run your evaluation set against both, then switch reads. Re-embedding a large corpus is one of the biggest one-off compute costs an AI platform incurs, so schedule and budget it like a release.

Analysing retrieval results on a laptop
Analysing retrieval results on a laptop

Permission-Aware Retrieval

The hard requirement: if a user cannot open a document in the source system, the assistant must not retrieve any chunk of it. Copy source ACLs into chunk metadata as principal IDs (users and groups), resolve the requesting user's groups at query time, and filter before ranking:

principals = directory.resolve(user_id)  # user + group ids
hits = index.search(
    vector=embed_query(q),
    filter={"acl": {"$in": principals}},
    top_k=50,
)
results = rerank(q, hits)[:8]

Never filter after the model has seen content. Sync ACL changes on the same cadence as content changes, and write tests that query as a denied user and assert zero hits.

Where should vectors live? For modest corpora, vector support in Postgres or your lakehouse keeps governance in one place. Move to a dedicated vector database when scale, latency or complex metadata filtering demand it.

Hybrid Search and Reranking

Pure vector similarity misses exact identifiers such as contract numbers and product codes. Combine keyword (BM25) and vector retrieval, then rerank a larger candidate set with a cross-encoder. This one change often improves answer quality more than switching generation models.

Evaluate Like You Mean It

Keep a versioned evaluation set of real questions with expected source documents and reference answers. Measure retrieval recall separately from answer quality, so you know which stage to fix. Run it on every change to parsing, chunking, embedding, prompts or models.

Observability for the Pipeline

Instrument it like any other production data system. Track documents processed, parse failures by file type, chunks per document, embedding latency and cost, index size, and the age of the oldest unsynced change per source. Log every retrieval with the query, the user's principals, the chunk IDs returned and whether the answer was rated useful. When quality drops, these logs tell you whether the cause is a stale source, a parsing regression or a ranking change, instead of leaving you to swap models and hope.

Structured Data Still Matters

Many questions are really metric questions ("what was churn last quarter?"). Route those to a semantic layer over governed tables rather than retrieving text, so agents get consistent, permission-checked numbers.

The broader platform, CDC, lakehouses, contracts, governance and cost, is covered in our guide to what a big data development company builds in 2026. For the application layer, see our LLM integration services.

Frequently Asked Questions

What is the most common cause of wrong RAG answers?

Stale or poorly parsed content in the index. Freshness, delete handling and structure-aware chunking usually matter more than the generation model.

Should ACL filtering happen in the vector store or the application?

In the retrieval query, before ranking and before the model sees anything. Application-side filtering after retrieval risks leaks and reduces recall.

How often should documents be re-embedded?

Only when content changes or you upgrade the embedding model. Track model version per vector so upgrades can run as a controlled, evaluated migration.

Do I need a dedicated vector database?

Not at first. Vector extensions in existing databases work well for modest corpora and keep governance simple. Switch when scale or filtering complexity requires it.

Subscribe to techcirkle