Serving frontier intelligence at scale is an industrial undertaking and until we build our own inference capacity, mere access to open-weight models will not be enough. We need to build the inference infrastructure and acquire the process knowledge needed to serve intelligence to those who need it.
This is a link-enhanced version of an article that first appeared in the Mint. You can read the original here. For a full archive of all my articles please visit my website.
Late last month, Moonshot AI released the weights for Kimi K3, a 2.8-trillion-parameter artificial intelligence (AI) model so advanced that it only lags the very best models from OpenAI and Anthropic. Even though it has been released as an open-weight model with weights available for download, most consumers will not be able to experience its frontier capabilities. Amid the hype around announcements of open-weight models, what often gets obscured is that model weights are only part of the story. Without inference and test-time compute, it is impossible to serve users frontier AI, no matter how powerful the underlying model might be.
When ChatGPT was launched, inference was a simple matter of running a query against the model’s weights and having it predict a response, one word at a time. This is no longer the case. Today’s models are designed as committees of specialist sub-networks (called a mixture-of-experts), only a handful of which are consulted for each query. This design ensures that not all weights are activated every time a query is presented. Kimi K3 is designed with 896 experts, of which only 16 are consulted for any given token, so that for every query, only 104 billion parameters (out of 2.8 trillion) do any work. This indicates that inference compute plays a far more significant role than it used to. And that frontier intelligence is no longer just about access to top models.
But it doesn’t stop there. Modern inference involves reasoning: writing out chains of questions, checking their conclusions, backtracking and trying again until an acceptable answer is reached. This process (which takes place behind the scenes every time a user asks a query) is why the answers we get are so much more accurate and relevant. In addition, AI models use a variety of tools (search engines to gather current facts, code to verify calculations, etc) to reduce hallucinations, ensure that the answers generated are up to date beyond the underlying model’s knowledge cut-off and remain logically consistent.
It is a combination of all these features that constitute the ‘frontier’ nature of the experience. All of which is to say that inference now goes far beyond running a query against the weights. It requires considerable computational resources (what engineers call test-time compute) and significant engineering expertise, without which even the most capable model will not deliver frontier intelligence.
Offering all of this at population scale is an industrial undertaking. In the first place, the weights need to be stored in the fastest, most expensive memory available, spread across dozens of accelerators and wired together to operate as a single computer. The maths is unforgiving. At 1.4 terabytes, K3’s weights need something in the order of 18 H100 chips just to sit in memory. Even after Moonshot AI engineered its way around the fact that longer conversations consume more memory, the weights alone need 18 accelerators before a single consumer is served.
In a mixture-of-experts architecture, this is even harder to implement, since it is hard to know in advance which specialists will be required for any given query. Finally, the reasoning and tool use needed to deliver frontier AI calls for a level of process knowledge that can only be learnt through the experience of delivering it at scale.
Having argued for years that the digital public infrastructure (DPI) approach is the way to build public systems, I can see how regular readers might expect me to argue in favour of applying that playbook to AI. If the same operating philosophy of open, modular infrastructure could be extended to AI, we might be able to achieve the same success with intelligence as we did with public services. This is harder than it appears, given the fundamental differences between the way software and AI stacks are designed.
DPI was able to transform the delivery of public services because the hard part was designing the protocol. Once the digital rails had been designed, it cost next to nothing to serve each additional user. This is why open, modular DPI tends to be more economical than vertically integrated software solutions. AI inverts that. Even though the rails are cheap—K3 gives frontier capability away for nothing—serving intelligence is expensive.
There can be no frontier intelligence without the process knowledge required to design and deliver test-time compute. It is far from trivial to understand just how much reasoning budget a query needs before the returns flatten; how to batch requests without starving those that need speed; when to keep a cache warm and when to throw it away; and how to engage experts without overloading memory. This knowledge is not bundled along with the open weights that are so easy to download. It has to be learned through trial and error by running these models for months at a stretch.
Hidden among the many announcements at Epoch 2026, Sarvam’s inaugural developer conference, was news that the company was launching inference services on Indian infrastructure. To me, this was the most significant announcement of the event. It indicates that we are finally taking inference seriously.
Open weights are not the same as open access. Until we learn how to serve intelligence on our own infrastructure, and on our own terms, K3 will just be a 1.4 terabyte file that we can download for free but cannot fully use.