How Transformers Work

The Core Transformer Architecture

The Transformer architecture, introduced by Vaswani et al. in 2017, replaced recurrent neural networks (RNNs) by processing entire sequences of tokens in parallel. This shift eliminated the sequence-length bottleneck and enabled training at unprecedented scale.

The model relies on stackable layers composed of two primary sub-layers: Multi-Head Self-Attention and Position-Wise Feed-Forward Networks (FFN).

graph TD
    Input["Token Embeddings + Positional Encoding"] --> MultiHead["Multi-Head Self-Attention"]
    MultiHead --> AddNorm1["Add & LayerNorm"]
    AddNorm1 --> FFN["Feed-Forward Network"]
    FFN --> AddNorm2["Add & LayerNorm"]
    AddNorm2 --> Output["Output Logits"]
    
    %% Residual Connections
    Input -.-> AddNorm1
    AddNorm1 -.-> AddNorm2

---

The Multi-Head Self-Attention Pipeline

Self-attention allows the model to score the relevance of other tokens in a sequence relative to a target token. The mathematical operation relies on projecting input vectors into Query ($Q$), Key ($K$), and Value ($V$) matrices:

1. Linear Projections: Input embeddings $X$ are multiplied by weight matrices $W^Q$, $W^K$, and $W^V$: $Q = XW^Q, \quad K = XW^K, \quad V = XW^V$

2. Scaled Dot-Product Calculation: We compute the dot product similarity between queries and keys, scale by the dimension size $\sqrt{d_k}$ to prevent gradient saturation, and apply a softmax function: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

3. Multi-Head Aggregation: Rather than performing attention once, multi-head attention projects the inputs into $h$ different subspaces, calculates attention in parallel, and concatenates the outputs: $\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$ where each head is calculated as: $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$

---

Layer Normalization and Residual Connections

To ensure stable gradient flow during deep network training, Transformers integrate residual connections (skip connections) around both major sub-layers. The output of each sub-layer is defined as:

$\text{SubLayerOutput} = \text{LayerNorm}(x + \text{SubLayer}(x))$

The Layer Normalization step normalizes values across the hidden dimension of each individual token, stabilizing activations independently of the batch size.

---

Position-Wise Feed-Forward Networks

Following attention normalization, each token’s representation passes through a Position-Wise Feed-Forward Network. This consists of two linear transformations with a non-linear activation (typically GeLU or SwiGLU) in between:

$\text{FFN}(x) = \text{max}(0, xW_1 + b_1)W_2 + b_2$

This FFN processes each token coordinate independently and identically, introducing non-linear capacity to model complex feature interactions.