Pipeline Parallelism

Technical Overview of Pipeline Parallelism

Pipeline Parallelism (PP) is a distributed training and execution model that partitions a neural network sequentially by assigning different blocks of layers to different devices.

For example, in a transformer model containing 80 layers, a Pipeline Parallelism setup with 4 GPUs (PP=4) assigns layers 1-20 to GPU 0, layers 21-40 to GPU 1, and so on. Since layers must execute sequentially, GPUs would normally sit idle waiting for previous activations. To prevent this, workloads are split into smaller micro-batches to enable concurrent execution.

Key Architecture & Implementation

To optimize GPU utilization and minimize idle phases (bubbles), developers implement scheduling strategies:

• 1F1B (One Forward, One Backward):
  • In training, each GPU alternates between executing one forward micro-batch and one backward micro-batch.
  • This stabilizes VRAM requirements because activations are cleared sooner compared to naive pipelines.
• Interleaved 1F1B:
  • Each GPU handles multiple non-contiguous layer subsets (e.g., GPU 0 handles layers 1-5 and 21-25), reducing the size of the idle bubble.

GPU 0 (L1-20)  --> F1 --> F2 --> B1 --> B2
GPU 1 (L21-40)      --> F1 --> F2 --> B1 --> B2

Core Parameters

• Bubble Overhead Ratio: The theoretical time fraction spent in idle bubbles is estimated as: ext{Bubble Fraction} = rac{PP - 1}{M} Where $PP$ is the pipeline depth and $M$ is the number of micro-batches per training step.
• Network Demands: Low compared to Tensor Parallelism. Only layer boundary activations are sent between GPUs, allowing PP to scale across nodes over standard networks.

Real-world Applications

• Scaling models with hundreds of billions of parameters across multiple physical server nodes (e.g., Llama-3-405B).
• Used in orchestration libraries like Megatron-LM, DeepSpeed ZeRO-3, and Colossal-AI.