
If you profiled your app's energy footprint the way you profile a hot loop, most of you would find something uncomfortable: the biggest cost center isn't the render path or the query planner. It's the inference call you bolted onto half your endpoints last quarter.
A conventional web request fetches rows and serializes them — cheap, cache-friendly, and optimized by decades of tooling. An inference request pushes a prompt through billions of parameters on power-hungry accelerators, and the energy per call can sit orders of magnitude above the database query right next to it. The catch is that this cost is invisible in your normal telemetry. It hides inside an API bill or a GPU reservation instead of showing up as a span in your traces. So this post treats inference as what it actually is — a first-class performance and cost surface — and walks the engineering levers that shrink it. Because compute is compute, every one of these also cuts your bill.
Stop thinking "UI green," start thinking "compute green"
The old green-software checklist — dark mode, image compression, lazy loading — optimizes the wrong layer for AI-heavy products. It's not wrong, it's just aiming at kilobytes while your inference path is spending joules.
A useful mental model: a green system minimizes three quantities simultaneously.
-
Energy per unit of useful work — the efficiency of each operation you actually need.
-
Useless work — computation that produces zero product value: polling for data that changes hourly, re-rendering unchanged views, recomputing cacheable results, shipping fields no client reads.
-
Carbon intensity of the electricity — a function of where and when the job runs, entirely outside your codebase.
Most teams over-index on (1) at the UI layer and never audit (2) or (3) at the compute layer, which is exactly backwards for anything running LLMs at scale.
Right-sizing the inference path
This is the highest-leverage work available. The principle: match the model and the compute to the job, and stop paying — in watts and dollars — for capability you don't use.
Model routing. Put a router in front of your model calls that classifies task difficulty and sends each request to the smallest capable model by default. Simple extraction, classification, and formatting rarely need a frontier model; reserve the expensive engine for genuinely hard reasoning. This single change often produces the largest single drop in both energy and spend.
Semantic caching. Identical and near-identical prompts should not re-trigger inference. An embedding-keyed cache in front of the model collapses a surprising fraction of production traffic — FAQ-style queries, repeated system prompts, common completions — into cache hits that cost effectively nothing.
Batching and optimized serving. Grouping concurrent requests and running them through a serving stack built for high utilization (continuous batching, paged attention, and similar) raises accelerator occupancy, so you burn far less energy per request than one-at-a-time calls on underused hardware.
Distillation and fine-tuning. A smaller model fine-tuned on your specific task frequently matches a giant general model at a fraction of runtime cost. Unlike caching, this permanently lowers the energy floor of the feature rather than dodging individual calls.
Prompt and context discipline. Tokens are the unit of both cost and compute. Tighter retrieval, trimmed context windows, and leaner system prompts reduce tokens processed per call, and it compounds across every request.
Done together, disciplined LLM integration is simultaneously the greenest and the cheapest way to run models in production. None of it is sustainability theater — each lever removes real FLOPs.

Don't retrain when you can adapt
Inference is the recurring cost; training is the spiky one. Every retrain-from-scratch pays the full energy bill again. For most product teams the discipline is: start from a capable base model, fine-tune sparingly, and retrain only when the data genuinely justifies it. A deliberate retraining cadence is one of the least-discussed and highest-impact decisions you'll make on the agentic and AI workflow side.
The infrastructure layer: region, schedule, right-size
Once the model layer is tuned, the cloud is where the remaining carbon lives — and it's the easiest place to cut without changing product behavior.
-
Region selection. The same workload in a region with a cleaner grid mix carries substantially less carbon. This is a deploy-target decision, not a code change.
-
Carbon-aware scheduling. Defer non-urgent batch work — reports, retraining, pipelines — to windows or regions where the grid is cleanest. Only background jobs are affected; nothing user-facing shifts.
-
Right-sizing fleets. Most clusters are over-provisioned. The delta between reserved and used capacity is carbon and spend with zero return. Autoscaling and serverless collapse that gap; idle accelerators are the worst offenders because they're both expensive and near-useless when parked.
Instrument it, or you're guessing
Intuition about energy hotspots is usually wrong — teams blame the front end and find a nightly job or a chatty AI endpoint. You don't need perfect carbon accounting to act. Spend and carbon track so tightly that your billing breakdown by service is a usable proxy: the most expensive service is almost always the most carbon-intensive one. Start from cost-per-service and fleet utilization, both already in your dashboards, then layer provider carbon APIs and per-call footprint estimators on top to refine. The goal is a directional signal good enough to rank the backlog, not a number good enough to publish.
Why this survives the next budget review
Efficiency initiatives that depend on goodwill get cut in the first tough quarter. Ones that also cut the cloud bill survive. In cloud-native systems carbon and cost are nearly the same variable — an over-provisioned fleet wastes both, an oversized model wastes both, a chatty API wastes both. Frame this work as reliability and margin engineering rather than a separate "green" program, put a standing slice of the backlog against it, and it becomes how the team builds by default instead of a thing someone has to remember to care about.
Want the strategic version of this argument for stakeholders? Read the full guide on TechCirkle.
Frequently Asked Questions
Where do I get the biggest single win in an AI app?
A model router. Sending each request to the smallest capable model by default, and reserving the frontier model for hard reasoning, typically produces the largest single drop in both energy and spend. Add semantic caching next — together they cut a large share of production inference load before you touch anything else.
How is semantic caching different from a normal cache?
A normal cache keys on exact input. A semantic cache keys on embeddings, so near-identical prompts — reworded questions, repeated system contexts — hit the same entry. That matters for LLM traffic, where users phrase the same intent dozens of ways and exact-match caching would miss almost all of it.
Does batching hurt latency?
Continuous batching in modern serving stacks adds minimal latency while sharply raising accelerator utilization, so you get lower energy per request without the tail-latency penalty of naive request queuing. For throughput-bound AI workloads it's usually a clear win on both axes.
Should I fine-tune a smaller model or just prompt a big one?
If a task is stable and high-volume, fine-tuning or distilling a smaller model often matches the large one at a fraction of runtime cost and permanently lowers the feature's energy floor. For low-volume or highly variable tasks, prompting a larger model with tight context is usually the pragmatic choice.
How do I measure per-call energy without a full carbon platform?
Start with what you already have: cost-per-service and fleet utilization. Because spend and carbon track closely, your most expensive service is almost certainly your most carbon-intensive one. Then add provider carbon APIs and open per-call estimators to sharpen the picture. Directional accuracy is enough to prioritize.
Is carbon-aware scheduling worth the engineering complexity?
For genuinely deferrable batch work — retraining, reporting, pipelines — yes, because you cut the carbon of identical computation with no product impact. For latency-sensitive, user-facing requests it usually isn't worth it; focus those on model routing, caching, and right-sizing instead.