Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Building a Product Catalogue Enrichment Pipeline That Doesn't Corrupt Your Data

How to build an AI product data enrichment pipeline with a staging layer, confidence scoring and human review — without corrupting your master catalogue.

techcirkle
5 min read
Cover image for Building a Product Catalogue Enrichment Pipeline That Doesn't Corrupt Your Data
Ecommerce runs on catalogue data that is almost always worse than its owners believe
Ecommerce runs on catalogue data that is almost always worse than its owners believe

Every AI commerce feature you want to build — semantic search, personalisation, agent-readiness, visual similarity — is bounded by the same constraint: how well your catalogue describes your products. And almost every catalogue is worse than its owners believe.

This is a build guide for fixing that with models, without creating a worse problem than the one you started with.

The state you are probably starting from

Run this query before anything else — attribute completeness by category:

SELECT category,
       COUNT(*) AS skus,
       AVG(CASE WHEN material   IS NOT NULL THEN 1.0 ELSE 0 END) AS material_fill,
       AVG(CASE WHEN colour     IS NOT NULL THEN 1.0 ELSE 0 END) AS colour_fill,
       AVG(CASE WHEN dimensions IS NOT NULL THEN 1.0 ELSE 0 END) AS dim_fill
FROM products
WHERE active = true
GROUP BY category
ORDER BY skus DESC;

The typical result is 30–50% fill on attributes that matter, wildly uneven across categories, and a long tail of near-empty ones nobody has looked at since the category launched. Alongside that: multiple vocabularies for identical concepts ("navy", "dark blue", "NAVY BLUE") because three suppliers feed the same category.

Humans compensate for this effortlessly. Retrieval systems and shopping agents do not — they take the data literally and conclude the product is unsuitable.

The architecture

Four stages. The third one is the one people skip, and skipping it is how you corrupt a catalogue.

Source data ──► Extraction ──► Staging (confidence-scored) ──► Promotion ──► Master catalogue
   (images,        (VLM +          (never the master           (auto or
    text,           text model)     table)                      human review)
    supplier feeds)                      │
                                         └──► Review queue ──► Evaluation set

Stage 1 — Extraction

Vision-language models handle attributes visible in product photography — material, pattern, closure type, neckline, fit, colour. Text models normalise supplier-supplied text onto your taxonomy and reconcile conflicts.

Design notes that matter in production:

  • Extract to a strict schema, not free text. Constrain output to your actual enum values per attribute. A model returning "navy blue" when your taxonomy has navy creates a normalisation problem you just moved downstream.

  • Ask for a confidence signal explicitly and treat it as advisory, not calibrated. Model self-reported confidence is loosely correlated with correctness at best — calibrate it against your own review outcomes before trusting thresholds.

  • Include the source evidence in the output — which image, which text span. Human reviewers are dramatically faster when they can see why a value was proposed.

  • Batch aggressively. Per-SKU cost is what makes a 200,000-item catalogue feasible or not.

Stage 2 — Staging

The non-negotiable rule: model output never writes to the master catalogue.

CREATE TABLE proposed_attributes (
  sku            TEXT NOT NULL,
  attribute      TEXT NOT NULL,
  proposed_value TEXT NOT NULL,
  confidence     REAL NOT NULL,
  model_version  TEXT NOT NULL,
  evidence       JSONB,
  extracted_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  status         TEXT NOT NULL DEFAULT 'pending',  -- pending|promoted|rejected
  reviewed_by    TEXT,
  reviewed_at    TIMESTAMPTZ,
  PRIMARY KEY (sku, attribute, model_version)
);

Why this matters concretely: without it you cannot distinguish asserted facts from inferred ones, you cannot roll back a bad extraction run, and you cannot tell whether a wrong value came from a supplier or a model. We have seen a team spend three weeks unpicking a single bad run that wrote directly to production. The staging table would have made it a one-line status update.

Stage 3 — Promotion policy

Simple and explicit:

  • Confidence above threshold and no conflicting asserted value → promote automatically

  • Confidence above threshold but conflicts with an existing value → human review, always

  • Confidence below threshold → review queue, prioritised by SKU revenue

  • Any attribute feeding a legal or safety claim → human review regardless of confidence

That last rule is worth stating explicitly. Material composition, allergen information, safety certifications, and regulatory claims should never auto-promote no matter how confident the model is. The downside is not a bad search result.

Set the threshold from your own review data, not a default. Sample a few hundred proposals across the confidence range, have someone label them, and pick the point where precision meets your tolerance.

Stage 4 — The evaluation loop

Every human review decision is labelled data. Route it back:

  • Rejected proposals become negative examples for prompt refinement

  • Category-level precision by attribute tells you where extraction is failing

  • Reviewer disagreement flags attributes where your taxonomy is ambiguous, which is a real and common finding — sometimes the model is not wrong, your schema is underspecified

Measure downstream, not coverage

The metric trap here is severe. "We populated 94% of attributes" is a vanity number — it says nothing about correctness or business impact.

Track instead:

  • Null-result search rate by category, before and after

  • Filter-refined session rate (customers can only filter on attributes that exist)

  • Add-to-cart rate from filtered results

  • Review precision by attribute and category

  • Return rate in categories where enrichment targeted fit or sizing attributes

The last one is the sleeper. In apparel especially, better structured sizing and fit data measurably reduces returns, and returns are where ecommerce margin quietly dies.

Operational data — where enrichment quality eventually shows up as margin
Operational data — where enrichment quality eventually shows up as margin

Why this is suddenly urgent

Catalogue quality used to be a slow-burn merchandising concern. It became urgent because a growing share of buyers are software.

Adobe measured AI-referred retail traffic up 393% year over year in Q1 2026, converting substantially better than traditional search. Those agents never see your product page — they read structured data and decide in milliseconds. A missing attribute is not an untidy record; it is a silent exclusion from consideration with no analytics event attached.

The catalogue is no longer a database. It is the interface.

Full architecture and rollout guidance: AI in Ecommerce: What Actually Drives Revenue in 2026.

Frequently Asked Questions

Which attributes should I extract first? The ones your customers filter on and your search queries reference, in your highest-revenue categories. Pull your top zero-result queries and your most-used filters — the intersection is your extraction backlog, ranked.

How accurate is vision-based attribute extraction really? Good on visually evident properties — colour, pattern, closure, neckline, broad material class. Weak on anything requiring domain knowledge not visible in the image, like precise fibre composition or certification status. Structure your promotion policy around that distinction rather than a single global threshold.

Should I re-run extraction when models improve? Yes, and keep model_version in the primary key so you can compare runs rather than overwrite them. Re-running on a newer model and diffing against prior proposals is also a cheap way to find categories where the old run was weak.

How do I handle multilingual catalogues? Extract to a canonical language and taxonomy, then translate for display. Extracting independently per locale produces divergent values for the same SKU, which is very difficult to reconcile later.

What team size does this need? An engineer to build the pipeline and a part-time reviewer with category knowledge. The reviewer matters more than people expect — extraction quality is bounded by whether someone can adjudicate ambiguous cases and clarify the taxonomy when it turns out to be the actual problem.

How long does a first pass take? Six to ten weeks for a mid-size catalogue, most of it spent on taxonomy definition and review tooling rather than model integration. The extraction itself is usually the easiest part of the project.


TechCirkle builds data pipelines and AI systems for commerce. See our AI development services or talk to us.

Subscribe to techcirkle