Unlocking Asynchronicity in Continuous Batching
Use CUDA streams to separate CPU batch preparation from GPU compute, launching H2D, compute, and D2H streams concurrently to eliminate 24 % idle GPU time.
Implement asynchronous batching by creating separate CUDA streams for H2D, compute, and D2H, ensuring proper stream synchronization, and benchmark GPU utilization to confirm ~24 % speedup.
Summary
The article explains how to separate CPU and GPU workloads in continuous batching to eliminate idle GPU time, a problem that can account for nearly 24 % of total runtime in synchronous batching. Continuous batching traditionally uses a single loop where the CPU prepares a batch, transfers it to the GPU, the GPU computes, and the CPU samples the output, causing the GPU to idle while the CPU updates and vice versa. By introducing asynchronous batching, the CPU can prepare the next batch while the GPU processes the current one, using separate CUDA streams for host‑to‑device (H2D), compute, and device‑to‑host (D2H) operations. The implementation requires non‑default CUDA streams, non‑blocking memory copies, and explicit stream synchronization so that compute waits for H2D and D2H waits for compute. No changes to model architecture or kernels are needed; the technique is applied in the transformers library and can be replicated in PyTorch. Benchmarking on an 8B model with 8 K tokens and batch size 32 shows a reduction from 300.6 s to 228 s, a 24 % speedup, confirming the effectiveness of the approach.
Key changes
- Synchronous continuous batching causes ~24 % GPU idle due to CPU‑GPU alternation
- Asynchronous batching uses separate CUDA streams: H2D, compute, D2H
- Non‑default streams allow GPU operations to run concurrently without CPU blocking
- Explicit stream synchronization ensures compute waits for H2D and D2H waits for compute
- No kernel or model changes required
- Implementation demonstrated in transformers library with PyTorch
- Speedup from 300.6 s to 228 s for 8 K tokens, batch size 32, 8B model