Continuous Batching

Technical Overview of Continuous Batching

Continuous Batching (also known as Iteration-level Batching) is an inference scheduling algorithm that groups active requests dynamically at the token iteration level.

Standard static batching groups multiple requests and processes them together. However, because requests have varying input and output lengths, the entire batch must wait for the longest request to finish generation, leaving GPU resources idle. Continuous Batching addresses this by dynamically adding and removing requests from the batch at each token generation step.

Key Architecture & Implementation

Unlike static batching, the scheduling loop in Continuous Batching executes at every step:

1. Pre-fill Phase: New requests are processed to generate their initial KV cache.
2. Decoding Steps: The scheduler executes a single token generation step for all active requests in the batch.
3. Dynamic Eviction: Requests that generate end-of-sequence (EOS) tokens are immediately removed from the batch.
4. Dynamic Insertion: Pending requests in the queue are inserted into the batch at the next step, utilizing the freed resources.

Static Batching:      [ Req A (50t) | Req B (10t) ] ---> Batch runs for 50 steps
Continuous Batching:  Step 10: Evict Req B, insert Req C ---> Batch size remains optimal

Core Parameters

• Latency Reduction: Significantly decreases queue time for requests.
• Throughput: Improves overall system serving throughput by up to 2x compared to static batching.

Real-world Applications

• Implemented in production serving engines like vLLM, Triton Inference Server, and TensorRT-LLM.

Why It Replaced Static Batching

Earlier LLM serving systems used static batching: a fixed group of requests was processed together, and the entire batch had to wait for the longest-running request in it to finish generating before any slot could be freed for a new request, even if shorter requests in the batch had completed generation much earlier. This wasted significant GPU capacity, since finished requests’ slots sat idle rather than being reused. Continuous batching (sometimes called iteration-level scheduling) instead makes batching decisions at every individual generation step rather than per full request: as soon as any request in the current batch finishes, a new waiting request can immediately take its place in the very next iteration, rather than waiting for the whole batch to complete. This keeps GPU utilization consistently high regardless of the wide variance in how long different requests take to finish, and it’s one of the core techniques (alongside PagedAttention) that serving engines like vLLM and TensorRT-LLM use to achieve dramatically higher throughput than naive batching approaches, particularly under mixed workloads where request lengths vary significantly.