Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Four of the Five Analytics Stack Layers Are Commodities Now

Ingestion, storage and BI have commoditised. The differentiated engineering sits in transformation and semantics — plan your budget accordingly.

techcirkle
5 min read
Cover image for Four of the Five Analytics Stack Layers Are Commodities Now
Data analyst reviewing a business analytics dashboard showing charts, metrics and KPIs
Data analyst reviewing a business analytics dashboard showing charts, metrics and KPIs

Tooling debates in analytics have gone quiet, and that is genuinely good news. The stack has five layers, there is now a defensible default at each, and — the part worth internalising — four of the five are commodity purchases.

That has a direct budgeting consequence: the differentiated engineering effort belongs in one and a half layers, and most proposals still distribute it as though all five were equally hard.

Layer 1 — Ingestion (commodity)

Managed connectors for anything standard. Fivetran, Airbyte, or the equivalent. Custom pipelines only where a connector genuinely does not exist — an internal service, a partner API nobody else integrates with, a legacy on-premise database with an unusual access pattern.

Paying a consultancy to hand-build a Salesforce or Stripe connector in 2026 is a signal worth acting on. It is billable hours against a solved problem, and it creates a maintenance liability you own forever.

The one place ingestion deserves real thought is schema drift and backfill behaviour. What happens when an upstream field changes type? Does the connector fail loudly, coerce silently, or drop the column? That answer matters more than the vendor choice.

Layer 2 — Storage and compute (commodity)

Snowflake, BigQuery, Databricks, or a lakehouse on open table formats. The differences are real but are rarely decisive for a mid-market analytics workload.

What is decisive is the cost controls you put around whichever you pick. Materialisation strategy, clustering keys, result caching, query monitoring with per-team attribution. Teams routinely spend three weeks evaluating warehouses and zero on the operational practices that determine the bill.

Open table formats have shifted the calculus somewhat — storing in Iceberg or Delta meaningfully reduces the cost of changing your mind later, which is worth something given how often organisations do.

Layer 3 — Transformation (differentiated)

dbt or SQLMesh. In version control. With tests. Running in CI.

This is the first layer where the choices are genuinely yours and genuinely consequential, because this is where your business logic gets encoded.

The anti-pattern to watch for: transformations living as saved queries inside the BI tool. It is extremely common, it happens gradually, and it means you do not have an analytics platform — you have a collection of untested logic with no review process, no history, and no way to know what breaks when a column changes.

Minimum bar for this layer:

models:
  - name: fct_orders
    columns:
      - name: order_id
        tests: [unique, not_null]
      - name: customer_id
        tests:
          - relationships:
              to: ref('dim_customer')
              field: customer_id

Uniqueness, referential integrity, freshness on sources, and anomaly detection on row counts. Everything else is refinement.

Team reviewing business growth data and financial charts together in a meeting
Team reviewing business growth data and financial charts together in a meeting

Layer 4 — Semantics (the differentiated half-layer)

This is the layer most stacks assembled in 2023 skipped entirely, and the reason many of them now behave badly under AI querying.

The semantic layer holds metric definitions declaratively — what active means, how revenue is recognised, which accounts are excluded — and applies them consistently regardless of who or what is asking.

metrics:
  - name: net_revenue
    description: Recognised revenue, USD, excluding internal transfers
    type: sum
    expr: amount_usd
    filters:
      - "is_internal = false"
      - "recognition_status = 'recognised'"

Before natural-language querying, you could get away without this because analysts carried the definitions in their heads. That worked, invisibly, for years. It stops working the moment a language model is answering questions directly, because the model resolves ambiguity by guessing rather than by knowing.

This is the layer you cannot buy pre-built. Every other layer ships with sensible defaults; this one encodes your specific business and has to be authored.

Layer 5 — Consumption (commodity, deliberately last)

BI tool, embedded analytics, notebooks, natural-language interface. Genuinely the least differentiated part of the stack in 2026, and the part that dominates most vendor demos.

Put it last on purpose. A consumption layer over a strong semantic layer is nearly interchangeable. A consumption layer over nothing is a liability with good typography.

What the distribution should look like

If you are reading a proposal, a rough sanity check on hours:

  • Ingestion: small, mostly configuration

  • Storage: small, mostly setup and cost guardrails

  • Transformation: the largest single block

  • Semantics: second largest, and rising

  • Consumption: modest, and it should be modest

Proposals that invert this — heavy dashboard construction, thin modelling — are pricing 2022 work at 2026 rates. The visible layer became the commodity. The invisible layer became the product.

The operational note

One consequence of getting layers 3 and 4 right: query volume goes up, sharply, because asking questions becomes cheap and people discover they can. Expect warehouse compute to climb thirty to sixty percent within two quarters of a successful rollout.

Treat that as a success indicator with a budget attached rather than a problem. But do attach the budget, and do assign someone ownership of materialisation and monitoring. It is a standing role, not a launch task.


The full buyer-side guide — cost ranges, the five disciplines inside an analytics engagement, build-versus-buy, and vendor evaluation questions — lives here: Data Analytics Services: A 2026 Guide for Technical Buyers.

TechCirkle builds data platforms and does LLM integration work for teams putting AI in front of their warehouses.

Frequently Asked Questions

Which layers of the analytics stack actually need engineering investment?

Transformation and semantics. Ingestion, storage and consumption have effectively commoditised, with sensible defaults available. Transformation and semantics are where your business logic gets encoded and cannot be bought pre-built.

Is it a problem if transformations live in the BI tool?

Yes. Saved queries inside a BI tool have no version history, no tests, and no review process. You end up with untested business logic and no way to predict what breaks when an upstream column changes.

What is the minimum test suite for a transformation layer?

Uniqueness and not-null on keys, referential integrity between fact and dimension tables, freshness checks on sources, and row-count anomaly detection on marts. Everything beyond that is refinement.

Why did semantic layers become important recently?

Analysts previously held metric definitions in their heads and applied them invisibly. Once a language model answers questions directly, that human buffer disappears and the model resolves ambiguity by guessing, so definitions must be declared and enforced.

Does the choice of warehouse matter much?

Less than most evaluations assume for a typical mid-market workload. The cost controls you build around it — materialisation strategy, clustering, query monitoring — affect the outcome far more than the vendor choice does.

How should hours be distributed in an analytics proposal?

Transformation should be the largest block, semantics second, with ingestion, storage and consumption comparatively small. A proposal weighted toward dashboard construction is pricing outdated work at current rates.

Why does compute spend rise after a successful rollout?

Because cheap questions get asked far more often, against more tables, at unpredictable times. A thirty to sixty percent rise within two quarters usually indicates genuine adoption, but it needs budget and an owner.

Subscribe to techcirkle