LLM Fine-Tuning Sizing Guide: Calculating LoRA, QLoRA, and Full Parameter Tuning Memory Demands

Introduction

Estimating GPU requirements for LLM fine-tuning is significantly more complex than sizing for inference. During inference, VRAM holds only the model weights and the KV cache. During training, the GPU must hold the weights, gradients, optimizer states, and forward-pass activation tensors.

This guide outlines how to calculate these training memory requirements to plan budgets and size clusters using LoRA, QLoRA, or Full Parameter tuning.

Sizing the Four Core Training Pools

The total VRAM requirement for model training is represented by:

$\text{Memory}_{\text{training}} = \text{Memory}_{\text{weights}} + \text{Memory}_{\text{gradients}} + \text{Memory}_{\text{optimizer}} + \text{Memory}_{\text{activations}}$

+-------------------------------------------------------------+
|                     Total VRAM for Training                 |
+--------------+---------------+----------------+-------------+
| Model Weights| Gradients     | Optimizer States| Activations |
| (Static)     | (1x Precision)| (AdamW Overhead)| (Dynamic)   |
+--------------+---------------+----------------+-------------+

1. Model Weights

As in inference, the weights consume:

$\text{Memory}_{\text{weights}} = P \times \frac{B}{8}$

Where $P$ is parameter count in billions and $B$ is the precision bit-width (typically 16-bit / 2 bytes for training).

2. Gradients

Gradients store the partial derivatives calculated during the backward pass. They are tracked for every trainable parameter and are typically stored in the same precision as training (usually 16-bit):

$\text{Memory}_{\text{gradients}} = P \times 2\text{ bytes}$

3. Optimizer States (AdamW)

The optimizer updates weights based on gradients. The industry standard AdamW optimizer tracks two states per parameter: the first momentum (mean) and the second momentum (uncentered variance).

In standard 32-bit AdamW, these states are stored in FP32 (4 bytes each), requiring 8 bytes per parameter:

$\text{Memory}_{\text{optimizer}} = P \times 8\text{ bytes}$

In 8-bit AdamW (e.g., using bitsandbytes), optimizer states are quantized, reducing the overhead to 2 bytes per parameter.

4. Activation Memory

Activation memory stores the intermediate output tensors calculated during the forward pass, which are required for gradient calculations during the backward pass.

Unlike other pools, activation memory is highly dynamic and scales linearly with batch size and sequence length.
To manage activation memory, engineers implement Activation Checkpointing (or Gradient Checkpointing), which discards intermediate activations and recalculates them on-demand during the backward pass. This reduces activation VRAM by up to 70% at the cost of a 30% execution time penalty.

Memory Profiles by Training Technique

graph TD
    A[Training Technique] --> B[Full Parameter Tuning]
    A --> C[LoRA]
    A --> D[QLoRA]
    
    B --> B1[Train 100% of Weights - Massive VRAM]
    C --> C1[Train <1% of Weights - Freeze Base - Mid VRAM]
    D --> D1[Train <1% of Weights - 4-bit Base - Low VRAM]

1. Full Parameter Tuning

Every parameter is updated. For a model trained in 16-bit precision using standard AdamW:

Weights: $2P$
Gradients: $2P$
Optimizer States: $8P$
Total Static Overhead: $12P$ (12 bytes per parameter)

A 7B model requires a minimum of $7 \times 12 = 84\text{ GB}$ of static VRAM before factoring in activations. This necessitates using distributed setups (e.g. FSDP, DeepSpeed Zero-3) even for small models.

2. LoRA (Low-Rank Adaptation)

Base model weights are frozen. Trainable parameters are restricted to low-rank adapter matrices (typically $<1\%$ of total parameters). For a 16-bit base model with $A$ trainable adapter parameters:

Base Weights (Frozen): $2P$
Active Weights (Adapters): $2A$
Gradients: $2A$
Optimizer States: $8A$
Total Static Overhead: $2P + 12A$

Since $A \ll P$ , the static memory overhead is close to the raw inference weight footprint ( $2\text{ bytes per parameter}$ ).

3. QLoRA (Quantized LoRA)

QLoRA quantizes the frozen base model weights to a special 4-bit format (NormalFloat4 or NF4) and executes training updates on 16-bit LoRA adapters:

Base Weights (Frozen, 4-bit): $0.5P$
Active Weights (Adapters, 16-bit): $2A$
Gradients: $2A$
Optimizer States: $8A$
Total Static Overhead: $0.5P + 12A$

A 7B model’s base weights are compressed to $3.5\text{ GB}$ , allowing full fine-tuning on a single consumer GPU (like an RTX 4090 with 24GB VRAM) with a large batch size.

Fine-Tuning Hardware Sizing Guide

Model Size	Method	Base Precision	Optimizer	Minimum GPU Hardware
7B	Full Tuning	BF16	FP32 AdamW	2x A100 (80GB)
7B	QLoRA	4-bit (NF4)	8-bit AdamW	1x RTX 4090 (24GB) or L4 (24GB)
70B	QLoRA	4-bit (NF4)	8-bit AdamW	2x A100 (80GB) or 2x H100 (80GB)
70B	Full Tuning	BF16	FP32 AdamW	2x 8x A100 (80GB) Nodes (DeepSpeed ZeRO-3)