Gradient Accumulation

Gradient accumulation is a technique that lets a model be trained with an effectively larger batch size than would otherwise fit in available GPU memory, by summing gradients across several smaller “micro-batches” before performing a single weight update. It’s one of the most common workarounds for GPU memory constraints in both pretraining and fine-tuning.

How It Works

Normally, a training step processes one batch of examples, computes gradients from that batch via backpropagation, and immediately updates the model’s weights using those gradients. With gradient accumulation, the optimizer instead processes several smaller micro-batches in sequence, computing and accumulating (summing) their gradients without updating weights after each one — only after a set number of micro-batches have been processed does the accumulated gradient get used for a single weight update, after which the accumulated gradient is reset to zero.

Why This Matters

Larger batch sizes generally produce more stable, less noisy gradient estimates, which can improve training stability and sometimes final model quality, and certain optimization schedules are tuned assuming a specific effective batch size. But larger batches also require more GPU memory to hold activations for the forward and backward pass. Gradient accumulation decouples these two concerns: it lets a team achieve the gradient quality of a large batch size (say, 512 examples) while only ever holding the memory footprint of a much smaller micro-batch (say, 32 examples) in memory at once, at the cost of taking more forward/backward passes — and therefore more wall-clock time — to complete one effective training step.

Practical Relevance

Gradient accumulation is a standard, low-risk lever when fine-tuning models on GPUs with limited memory: increasing the accumulation steps while decreasing the per-device micro-batch size preserves the same effective batch size and training dynamics while fitting within a smaller memory budget.

Interaction With Learning Rate and Batch Normalization

An important practical detail is that gradient accumulation changes the effective batch size a training run uses, which means learning rate and other batch-size-dependent hyperparameters should generally be adjusted to match, following established scaling rules (commonly scaling the learning rate proportionally with batch size) — simply adding gradient accumulation to an existing training configuration without revisiting these related hyperparameters can produce worse results than either the original small-batch or an intentionally-tuned large-batch configuration would. This interaction is one of the more common sources of confusion for practitioners newer to distributed training, where gradient accumulation is added purely to fit within memory constraints without appreciating that it also changes the effective training dynamics that other hyperparameters were tuned around.