Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Data Gravity and GPU Scarcity: Two Constraints That Decide Your Architecture

Two constraints now dominate cloud architecture for AI workloads, and both must be resolved before you choose regions or write a landing zone.

techcirkle
6 min read
Cover image for Data Gravity and GPU Scarcity: Two Constraints That Decide Your Architecture
Data centre network systems
Data centre network systems

Region selection used to be a three-variable problem: latency to users, data residency obligations, and price. Pick the region that balances them, done.

Two constraints now sit above all three, and both have an unpleasant property — they are effectively irreversible once you commit. Get them wrong and the remedy is a second migration.

Constraint one: data gravity

The principle is old. What changed is the magnitude.

Compute is portable. You can redeploy a service to a different region in an afternoon. Petabytes are not portable in any practical sense — moving them costs real money, takes real time, and is something most organisations do exactly once.

So once a large dataset is resident in a provider and region, every application that reads it is anchored there. Not by architecture, by economics. Your diagrams can claim portability; the egress bill and the latency budget decide otherwise.

What changed with AI workloads: read patterns went from occasional to continuous.

A traditional application reads a record when a user requests it. An embedding pipeline reads the entire corpus, then reads it again when you change chunking strategy, and again when you switch embedding models. A retrieval system reads continuously, for every query, forever.

If your corpus sits in provider A and your model endpoint in provider B, you pay egress on every one of those passes. And unlike most infrastructure costs, that line grows with adoption rather than staying flat. A successful feature becomes proportionally more expensive.

The practical rule: identify your gravity wells first — usually the transactional store, the document corpus, and the event stream — decide their home, and place compute around them. Not the reverse.

Teams that migrate applications first and sort out data later routinely end up with one logical system split across two providers, with data crossing the boundary continuously. The fix is a second migration nobody budgeted for.

Constraint two: accelerator availability

This one catches people genuinely off guard, because it contradicts the core cloud promise.

The implicit assumption behind twenty years of cloud architecture is that compute is available on demand, everywhere, in whatever quantity you want. For general-purpose instances that remains broadly true.

For current-generation accelerators it is not. Capacity is allocated, queued, and during periods of scarcity effectively rationed. A region you selected for excellent latency reasons may have months of lead time for reserved capacity, while a region two network hops further away has availability now.

That is an architectural constraint, not a procurement footnote, and it belongs in the design rather than surfacing as a surprise in month nine when the platform team tries to provision.

Engineer working with infrastructure
Engineer working with infrastructure

What to do about it:

  • Confirm availability in target regions before committing to a landing zone design, and get lead times in writing rather than verbally.

  • Separate inference placement from application placement. Inference often sits in a different region from the app tier without harming user experience, provided the data path is designed for it deliberately rather than by accident.

  • Keep a managed-endpoint fallback for burst demand rather than sizing reserved capacity for your peak. Over-provisioning reserved accelerators is usually the more expensive of the two available mistakes.

  • Model reserved versus on-demand at realistic utilisation. The break-even point is frequently higher than vendor calculators suggest, because those calculators assume a utilisation rate you will not hit in the first year.

Where the two constraints collide

Here is the tension that makes this genuinely hard rather than merely annoying.

Data gravity says: put compute next to your data. Accelerator scarcity says: put compute where capacity exists. When those two locations differ — and they frequently do, because accelerator capacity concentrates in a handful of large regions — you have a real architectural decision rather than a best practice to follow.

There is no universal answer, but there is a useful way to frame it. Ask which of your data flows are bulk and periodic versus small and continuous.

Bulk periodic flows — training runs, full-corpus embedding passes, batch analytics — tolerate distance badly on cost but fine on latency. If these dominate, keep compute close to data and accept the region constraint.

Small continuous flows — a retrieval query pulling eight chunks, an inference call with a few thousand tokens — tolerate distance fine on cost but badly on latency. If these dominate, you have more freedom to separate inference from storage, provided you account for the round trip.

Most real systems have both, which means the answer is usually to split: bulk processing next to the data, interactive inference wherever capacity and latency work out.

The third thing, since you are already here

Once you have settled placement, the remaining large cost lever is sequencing — and it is worth stating because it is invisible until the bill arrives.

Egress charges have no on-premises equivalent. Internal chatter in your own data centre is free in any sense that reaches a budget. Split that architecture across a boundary and the same traffic becomes metered.

The standard advice to migrate application by application is what maximises this exposure. It separates tightly coupled systems for the entire duration of the hybrid period, which reliably runs longer than planned.

Migrate in dependency clusters instead — groups of systems that talk to each other frequently, moved together. Larger waves, harder scheduling, materially lower cost and fewer boundary-straddling integration risks.

What to settle before writing the landing zone

Four things, in this order:

  1. Gravity wells and their home region. Near-irreversible; everything else accommodates it.

  2. Accelerator availability and lead times, confirmed in writing for those regions.

  3. Inference placement — same region as data, or separated with the data path designed for it.

  4. Wave sequencing by dependency cluster, derived from observed telemetry rather than interviews.

Then write the landing zone, and put the AI guardrails in it now rather than in a later pass: model endpoints restricted to approved providers and regions, outbound allowlists for anything running agent tooling, and classification metadata travelling with datasets. All cheap at design time, all awkward to retrofit across a populated estate.

Frequently Asked Questions

Why does data gravity matter more with AI workloads?

Because read patterns shifted from occasional to continuous and bulk. Embedding pipelines do full-corpus passes and retrieval systems read on every query, so cross-region data paths generate costs that scale with adoption.

Is accelerator scarcity a real planning constraint?

Yes. Availability varies by region and lead times for reserved capacity can run to months. Confirm it before committing to a landing zone rather than assuming on-demand availability.

Can inference run in a different region from the data?

Often yes, for small continuous flows like retrieval queries, provided the round trip is accounted for. Bulk periodic flows such as training and full-corpus embedding should stay close to the data.

How do I decide between reserved and on-demand accelerator capacity?

Model it at realistic first-year utilisation rather than vendor-assumed utilisation. Keeping a managed-endpoint fallback for burst demand is usually cheaper than sizing reserved capacity to peak.

What is the most expensive sequencing mistake?

Migrating application by application, which separates tightly coupled systems across a metered boundary for the whole hybrid period. Dependency-cluster waves reduce that exposure substantially.

What should be decided before the landing zone is written?

Data placement, accelerator availability, inference placement, and wave sequencing. All four are expensive or impossible to change after workloads have landed.


Full article covering the seven Rs, cutover patterns, and post-migration FinOps: Cloud Migration Best Practices for AI-Era Workloads.

We help teams make these placement calls before they harden — AI development services and cloud application development.

Subscribe to techcirkle