Grouped-Query Attention (GQA)

Technical Overview of GQA

Grouped-Query Attention (GQA) is an attention mechanism that groups query heads together to share single Key-Value (KV) projection heads.

Autoregressive LLM generation requires caching attention Key-Value states (the KV cache) for previous tokens to avoid redundant calculations. As context lengths scale, storing these states consumes significant VRAM. GQA addresses this by allowing multiple query heads to share a single KV head, maintaining model reasoning capabilities while reducing the memory footprint of the KV cache.

Key Architecture & Implementation

GQA bridges the gap between Multi-Head Attention (MHA) and Multi-Query Attention (MQA):

• MHA: Every Query head has a dedicated Key and Value head ($H_Q = H_{KV}$). This is memory-intensive but offers maximum representational capacity.
• MQA: All Query heads share a single Key and Value head ($H_{KV} = 1$). This minimizes memory use but degrades model reasoning quality.
• GQA: Query heads are divided into groups, and each group shares a single Key and Value head ($1 < H_{KV} < H_Q$).

MHA:  (Q1 -> KV1), (Q2 -> KV2), (Q3 -> KV3), (Q4 -> KV4)
GQA:  (Q1, Q2 -> KV1), (Q3, Q4 -> KV2)
MQA:  (Q1, Q2, Q3, Q4 -> KV1)

Core Parameters & Sizing

• Group Size ($G$): The number of Query heads sharing a single KV head, defined as: G = rac{H_Q}{H_{KV}}
• KV Cache Reduction: Memory usage scales down by a factor equal to the group size $G$. For example, a group size of 8 reduces KV cache VRAM requirements by 8x.

Real-world Applications

• Used in modern open-weight LLMs like Llama 3, Mistral, and Gemma 2 to support long context windows (up to 128k tokens).