Pipeline Parallelism

Technical Overview of Pipeline Parallelism

Pipeline Parallelism (PP) is a distributed training and execution model that partitions a neural network sequentially by assigning different blocks of layers to different devices.

For example, in a transformer model containing 80 layers, a Pipeline Parallelism setup with 4 GPUs (PP=4) assigns layers 1-20 to GPU 0, layers 21-40 to GPU 1, and so on. Since layers must execute sequentially, GPUs would normally sit idle waiting for previous activations. To prevent this, workloads are split into smaller micro-batches to enable concurrent execution.

Key Architecture & Implementation

To optimize GPU utilization and minimize idle phases (bubbles), developers implement scheduling strategies:

• 1F1B (One Forward, One Backward):
  • In training, each GPU alternates between executing one forward micro-batch and one backward micro-batch.
  • This stabilizes VRAM requirements because activations are cleared sooner compared to naive pipelines.
• Interleaved 1F1B:
  • Each GPU handles multiple non-contiguous layer subsets (e.g., GPU 0 handles layers 1-5 and 21-25), reducing the size of the idle bubble.

GPU 0 (L1-20)  --> F1 --> F2 --> B1 --> B2
GPU 1 (L21-40)      --> F1 --> F2 --> B1 --> B2

Core Parameters

• Bubble Overhead Ratio: The theoretical time fraction spent in idle bubbles is estimated as: ext{Bubble Fraction} = rac{PP - 1}{M} Where $PP$ is the pipeline depth and $M$ is the number of micro-batches per training step.
• Network Demands: Low compared to Tensor Parallelism. Only layer boundary activations are sent between GPUs, allowing PP to scale across nodes over standard networks.

Real-world Applications

• Scaling models with hundreds of billions of parameters across multiple physical server nodes (e.g., Llama-3-405B).
• Used in orchestration libraries like Megatron-LM, DeepSpeed ZeRO-3, and Colossal-AI.

The Pipeline Bubble Problem

A naive implementation of pipeline parallelism suffers from “bubbles” — idle time where later stages in the pipeline sit waiting for the first micro-batch to work its way through earlier stages before they have anything to process, similar to how an assembly line’s later stations sit idle until the first item reaches them. Techniques like micro-batching (splitting a training batch into many smaller pieces that flow through the pipeline in a staggered, overlapping fashion) reduce this idle time significantly, though some bubble overhead is generally unavoidable. Pipeline parallelism is frequently combined with other parallelism strategies (data parallelism across pipeline replicas, tensor parallelism within each pipeline stage) in a technique called 3D parallelism, which is how the largest models are typically trained across thousands of GPUs simultaneously, since no single parallelism strategy alone scales efficiently to that scale on its own.