# The Case for Disaggregated Inference *Yarco Hayduk* By [Pragma Ventures](https://paragraph.com/@pragmaooo) · 2026-06-23 llm, ai, inference, serving, disaggregated, gpu --- Why Serving Architecture Matters -------------------------------- While the public conversation surrounding large language models focuses heavily on parameter counts, reinforcement learning techniques, and emergent reasoning capabilities, far less attention goes to the infrastructure required to serve them at scale. Operating LLMs under strict latency budgets is already a major infrastructure challenge, and agentic workloads further exacerbate the problem by multiplying both the volume of tokens flowing through these systems and the associated processing costs. Although model-level optimizations still play a meaningful role in improving runtime efficiency, inference economics increasingly depend on how LLM infrastructure allocates, shares, and schedules GPUs. The core challenge in LLM serving is that generating text is not a single, uniform workload. Inference runs in two very different phases, prefill and decode. Prefill parallelizes well across the prompt and tends to be limited by compute throughput, while decode is sequential and much more sensitive to memory bandwidth and tail latency. Their different bottlenecks suggest that they could share a GPU efficiently, but in practice prefill can prevent decode from running at the same time. Two Phases, Two Bottlenecks --------------------------- The prefill phase processes the full prompt to build the initial key-value (KV) cache. Decode then reads from and extends that cache as it generates the response. In both phases, the model represents tokens as vectors and passes them through the same sequence of layers, each containing trained weight matrices. The calculations each phase performs and the data it must fetch from memory determine the demands it places on the GPU. During prefill, all prompt tokens are available at once, so the model can process them in parallel at each layer. The model arranges the token vectors into a matrix, with one row for each token. At each layer, the GPU multiplies this token matrix by the layer's weight matrices using large matrix-matrix multiplications (GEMM kernels). Each weight value read from high-bandwidth memory (HBM) is reused across calculations for every token vector in the matrix, so the GPU performs many floating-point operations (FLOPs) for each byte it reads. This high arithmetic intensity generally makes prefill compute-bound for long prompts, although short prompts may not provide enough work to saturate the GPU unless several are processed together, a technique called batching. The catch is that prefill for a long prompt can monopolize the GPU, potentially stalling many shorter prefills and decode steps queued behind it. Once the prompt has been processed and the KV cache built, decode becomes inherently sequential, generating the output one token at a time. The model cannot start working on the next token until the current one is produced and added to the response generated so far, so a single request no longer offers the prompt-level parallelism that prefill did. For each new token in an individual request, the GPU reads the model weights and the KV cache built so far from HBM, then runs mostly small matrix-vector operations, often implemented as GEMV kernels, appending the new token's own key and value to the cache as it goes. Each step reads a large amount of data but performs only a small amount of computation on it, so the GPU's execution units are often stalled, waiting for that data to arrive. As a result, decode has low arithmetic intensity and is constrained more by memory bandwidth than by computation. End users feel this split whenever a model streams a response. They wait once for prefill to work through the prompt, a delay measured as time-to-first-token (TTFT), then wait again between each token as decode generates it, measured as time-per-output-token (TPOT). The hard part is lowering both delays at once, because TTFT is driven by compute-heavy prefill while TPOT is driven by memory-bound decode. When Prefill Blocks Decode -------------------------- Intuition says that mixing both phases on the same GPU should help utilization, since prefill is limited mainly by compute while decode is limited mainly by memory bandwidth. That only helps if the two phases can run concurrently, with each using capacity the other leaves idle. In practice, the large kernels used for prefill occupy most of the GPU, leaving too little room for decode kernels to run at the same time. Decode therefore waits even when prefill leaves memory bandwidth available. When many requests are scheduled on the same GPU, a long prefill can then delay the much smaller decode steps waiting behind it. Systems engineers would classify this delay as head-of-line blocking. For example, a single 4,000-token summarization prompt can occupy the GPU far longer than a single decode step, leaving active streams waiting on their next token. In production, the latency impact is easy to recognize. Active users get sudden mid-stream pauses when an incoming prefill stalls the execution queue. Tail latency becomes hard to control, so bursty traffic can cause systems to miss service-level objectives (SLOs), such as a time-per-output-token target below 50 ms. To avoid those SLO misses, the scheduler must balance minimizing first-token latency for new requests against preventing mid-stream pauses in responses already being streamed to users. Chunking Prefill to Protect Decode ---------------------------------- Most engines first use chunked prefill to limit how long prompt processing can block decode without adding GPU resources. Prefill and decode remain on the same GPU, but instead of processing a 4,000-token prompt in one uninterrupted prefill pass, the scheduler divides it into pieces capped at a configurable number of tokens and interleaves them with the decode steps of active requests. Although a long prompt may still take hundreds of milliseconds to process, scheduling decode after each prefill chunk limits the longest visible pause caused by prefill for active streams to roughly the time required to process that chunk. The model still processes each prompt token through the same layers, but every chunk is submitted to the GPU separately, adding scheduling and kernel-launch overhead and potentially reducing efficiency because each pass processes fewer tokens. Chunked prefill gives decode more chances to run, but it does not remove the underlying resource conflict, since both phases still compete for the same compute and bandwidth on the same GPU. Under long contexts or strict latency constraints, chunking introduces a different tradeoff. The scheduler must then choose how much of the prompt to process before yielding to decode again. Small chunks give active streams more frequent decode turns and reduce mid-stream pauses, but the incoming prompt needs more rounds to finish prefill, so time-to-first-token gets worse. Conversely, large chunks get the first token out sooner, but each chunk can still block active streams long enough for users to notice. Chunk-size tuning reduces head-of-line blocking, but it leaves prefill and decode on the same GPUs with the same parallelism configuration. Changes to GPU allocation or parallelism apply to both phases even when only one is the bottleneck, preventing operators from tuning or scaling that phase independently. Splitting Prefill and Decode Apart ---------------------------------- Instead of running both phases on the same GPUs, disaggregated inference assigns prefill and decode to separate GPU pools. Each pool consists of one or more GPUs dedicated to a single phase, so operators can configure the pools separately and add GPUs to whichever one is the bottleneck. Incoming prompts go to the prefill pool, which can process long prompts while the decode pool continues generating tokens for ongoing responses. Once prefill builds the KV cache, the decode pool takes over and runs the sequential token-generation loop against that cached state. This removes the shared queue between large prefill jobs and small decode steps, but it introduces a new potential bottleneck. The KV cache produced in the prefill pool has to be transferred to the decode pool before generation can continue. KV cache size scales with prompt length and depends on model architecture. Llama 2 70B requires about 328 KB of FP16 KV state per prompt token, so a 4,000-token prompt produces roughly 1.3 GB. The key question is whether the time spent moving that cache is smaller than the time saved by no longer making prefill and decode contend over one GPU. The answer depends heavily on the interconnect. To isolate the interconnect cost, assume the entire cache is transferred from the prefill pool to the decode pool over a single interconnect. The numbers below are raw lower bounds based on nominal one-way bandwidth, since protocol and software overheads make a real transfer over that path slower. Over PCIe Gen4 x16, at around 32 GB/s, moving 1.3 GB takes roughly 41 ms. PCIe Gen5 x16 doubles that bandwidth and brings the transfer down to roughly 20 ms. GPUDirect RDMA over RoCE (RDMA over Converged Ethernet) moves the cache directly between GPU memories across the network. On a 400GbE-class path, around 50 GB/s at line rate, the raw transfer time lands near 26 ms. NVLink, NVIDIA's high-bandwidth GPU-to-GPU fabric inside a server, gives each H100 SXM GPU 900 GB/s of bidirectional bandwidth into the NVSwitch fabric of an HGX H100 system. As a simplified one-way estimate, that is roughly 450 GB/s, which puts the same transfer at about 2.9 ms. Since a decode step is often expected to finish within tens of milliseconds, transferring the cache over PCIe or 400GbE can consume a meaningful fraction of the time it takes to generate the next token. Over NVLink, the transfer takes only a few milliseconds. These numbers describe the simple case where decode waits for the entire cache to arrive. Systems that support layer-wise KV transfer can reduce this exposed delay by transferring each layer's KV cache as soon as prefill produces it. KV cache data from earlier layers can transfer while prefill computes later layers, reducing the transfer delay between the end of prefill and the start of decode. Not every deployment has the NVLink topology, however. If the KV cache travels over slower PCIe links or across the network, decode waits longer before it can start, making disaggregation harder to justify. We saw the same tradeoff in our earlier work on heterogeneous actors for CPU/GPU scheduling. For small inputs, sending work from the CPU to the GPU meant paying setup and data-movement costs without giving the GPU enough parallel work to justify the transfer cost. That made keeping the processing on the CPU the better choice for those smaller workloads. For larger inputs, CPU execution took much longer, while setup and data movement became a smaller share of total runtime. The GPU path was therefore faster overall despite the extra data movement. LLM serving follows the same logic. Chunking prevents a long prefill job from blocking decode for its full duration, but each chunk still occupies the shared GPU and can delay the next token for active streams. When SLOs leave little room for that remaining interference, moving prefill to a separate GPU pool can remove enough waiting to justify the KV cache transfer. Short prompts under light load create much less interference, so the handoff may add latency without removing much delay. For those workloads, chunked prefill on a single GPU is often the better tradeoff. Research and Serving Frameworks for Disaggregated Inference ----------------------------------------------------------- Early research systems showed that separating prefill from decode could improve serving latency and throughput, and serving frameworks soon began incorporating the same approach. * [**DistServe (OSDI 2024)**](https://arxiv.org/abs/2401.09670)**:** One of the first papers to formalize prefill-decode disaggregation. DistServe splits prefill and decode into separate GPU pools, then co-optimizes resource allocation and parallelism for TTFT and TPOT targets. It reports up to 7.4× as many requests, or 12.6× tighter SLOs, compared with state-of-the-art serving systems while keeping more than 90% of requests within latency constraints. DistServe reports goodput because it measures throughput that remains within latency targets rather than raw tokens per second. * [**Splitwise (Microsoft, ISCA 2024)**](https://arxiv.org/abs/2311.18677)**:** Splitwise approaches the same prefill-decode split from a hardware-specialization angle. It separates prompt computation and token generation onto different machines, provisions each phase independently, and optimizes clusters for throughput, cost, and power. It reports 1.4× higher throughput at 20% lower cost than the evaluated baseline designs, or 2.35× more throughput with the same cost and power budgets. * [**vLLM**](https://docs.vllm.ai/en/latest/features/disagg_prefill/), [**SGLang**](https://docs.sglang.ai/advanced_features/pd_disaggregation.html), and [**NVIDIA Dynamo**](https://docs.nvidia.com/dynamo/design-docs/disaggregated-serving): Serving frameworks support both single-GPU chunking and explicit prefill-decode separation. vLLM enables chunked prefill by default whenever possible in V1 and documents experimental disaggregated prefilling with separate prefill and decode instances connected through KV cache transfer connectors. SGLang lets operators tune chunk size directly and supports prefill-decode disaggregation with Mooncake and NIXL transfer engines. NVIDIA Dynamo supports disaggregated serving with KV-aware routing and independently scalable prefill and decode worker pools. * [**Mooncake (Moonshot AI)**](https://arxiv.org/abs/2407.00079)**:** Mooncake centers the serving architecture around the KV cache itself. It separates prefill and decode clusters, uses GPU HBM, host DRAM, and SSD resources as a distributed KV cache layer, and schedules around KV cache reuse, transfer, and cluster overload. It reports up to a 525% throughput increase in simulated scenarios while meeting SLOs, and 75% more requests under real workloads. Routing and Hardware Placement in Disaggregated Inference --------------------------------------------------------- Once the phases are split, performance also depends on how the scheduler routes requests and assigns work across the available hardware, starting with locality. If several requests share a system prompt or document prefix, routing them to a prefill worker that already holds that prefix can avoid recomputing it. When the runtime also supports decode-side cache reuse, choosing a decode worker that holds matching cache blocks can avoid retransferring those blocks. SGLang's RadixAttention organizes cached prefixes in radix trees, while its cache-aware router tracks which workers are likely to hold reusable state. NVIDIA Dynamo applies similar cache-aware routing to prefill workers at data-center scale, with a planner that changes the size of the prefill and decode pools as the workload becomes more prefill-heavy or decode-heavy. Beyond locality, schedulers can improve efficiency by matching work to device capabilities, especially at the edge, where a distributed inference cluster may combine devices with very different compute and bandwidth profiles. Uniform partitioning across a heterogeneous cluster can make the whole system run at the speed of the weakest device. Capability-aware partitioning avoids that problem by giving each device a share of the work proportional to what it can actually handle, so no single machine becomes the bottleneck. The same placement problem arises in data centers that use a mix of H100, A100, and L40S GPUs. Prefill can make fuller use of the most compute-capable GPUs, while decode may leave more of their compute capacity underused. As a result, the scheduler must account for those differences instead of treating every GPU as interchangeable. When Disaggregation Pays Off ---------------------------- Choosing between chunked prefill and disaggregation requires comparing the waiting caused when prefill and decode share GPUs with the time required to transfer KV caches between separate pools. That comparison depends on how frequently new prompts arrive while decode is generating tokens for existing requests, how long those prompts take to process in prefill, how large their KV caches are for the chosen model, and how quickly the interconnect moves them. Once the phases are separated, cache-aware routing can preserve prefix reuse, while operators can choose which GPU types run each phase and scale each pool to its own demand. Compared with chunked prefill on shared GPUs, disaggregation is justified when it serves more requests without missing either latency target. At production scale, where latency and cost are tightly constrained, the serving architecture shapes both at a scale comparable to many model-level improvements, making serving architecture a first-order design decision. References ---------- * Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI 2024), [https://arxiv.org/abs/2401.09670](https://arxiv.org/abs/2401.09670) * Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024), [https://arxiv.org/abs/2311.18677](https://arxiv.org/abs/2311.18677) * Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, with RadixAttention (NeurIPS 2024), [https://arxiv.org/abs/2312.07104](https://arxiv.org/abs/2312.07104) * Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST 2025), [https://arxiv.org/abs/2407.00079](https://arxiv.org/abs/2407.00079) * Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models (2023), [https://arxiv.org/abs/2307.09288](https://arxiv.org/abs/2307.09288) * NVIDIA, NVIDIA Hopper Architecture In-Depth, [https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) * NVIDIA, Introducing NVIDIA HGX H100: An Accelerated Server Platform for AI and High-Performance Computing, [https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/](https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/) * vLLM Project, Optimization and Tuning: Chunked Prefill (v0.20.2), [https://docs.vllm.ai/en/v0.20.2/configuration/optimization/](https://docs.vllm.ai/en/v0.20.2/configuration/optimization/) * vLLM Project, Disaggregated Prefilling (experimental, v0.20.2), [https://docs.vllm.ai/en/v0.20.2/features/disagg\_prefill/](https://docs.vllm.ai/en/v0.20.2/features/disagg_prefill/) * SGLang Project, PD Disaggregation (v0.5.8 source snapshot), [https://github.com/sgl-project/sglang/blob/v0.5.8/docs/advanced\_features/pd\_disaggregation.md](https://github.com/sgl-project/sglang/blob/v0.5.8/docs/advanced_features/pd_disaggregation.md) * SGLang Team, SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs, [https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/](https://www.lmsys.org/blog/2024-12-04-sglang-v0-4/) * NVIDIA, Dynamo Disaggregated Serving (v0.9.1), [https://docs.nvidia.com/dynamo/v-0-9-1/design-docs/disaggregated-serving](https://docs.nvidia.com/dynamo/v-0-9-1/design-docs/disaggregated-serving) * NVIDIA, Dynamo Planner Guide (v0.9.1), [https://docs.nvidia.com/dynamo/v-0-9-1/components/planner/planner-guide](https://docs.nvidia.com/dynamo/v-0-9-1/components/planner/planner-guide) * Hayduk et al., Enhanced Energy Efficiency with the Actor Model on Heterogeneous Architectures (DAIS 2016), [https://inria.hal.science/hal-01434796v1](https://inria.hal.science/hal-01434796v1) --- *Originally published on [Pragma Ventures](https://paragraph.com/@pragmaooo/disaggregated-inference)*