Multi-Query Attention (MQA)

Technical Overview of MQA

Multi-Query Attention (MQA) is an attention mechanism where all Query heads share a single Key-Value (KV) projection head.

During LLM generation, storing the Key-Value projection vectors for preceding tokens (the KV cache) consumes significant GPU memory, limiting concurrent serving capacity. MQA addresses this memory bottleneck by forcing all query heads to share a single Key-Value head, reducing the memory footprint of the KV cache at the cost of a slight reduction in model reasoning capacity.

Key Architecture & Implementation

In Multi-Head Attention (MHA), each query head has a dedicated Key and Value head. In contrast, MQA projects the Key and Value matrices only once:

• Query Projection Shape: $[Batch, SeqLen, Heads_Q, Dim]$
• Key/Value Projection Shape: $[Batch, SeqLen, 1, Dim]$

During the attention step, the single KV representation is broadcast across all Query heads to calculate attention logits.

[ Query Head 1 ] ---\
[ Query Head 2 ] ----+---> [ Shared KV Head ]
[ Query Head 3 ] ---/

Core Parameters

• Memory Savings: Reduces the VRAM footprint of the KV cache by a factor equal to the number of query heads (e.g., a 32x reduction for a model with 32 query heads).
• Throughput Increase: Frees up VRAM, allowing larger batch sizes and higher throughput during serving.

Real-world Applications

• Used in early LLMs like Falcon and Google’s PaLM to optimize serving efficiency.