Multi-head self-attention: mechanics and motivation

What's actually being asked

Explain multi-head self-attention — how each head independently projects Q/K/V, computes attention, and how the outputs get concatenated and projected — and why multiple heads help.

Answer

The mechanism

A single attention head has only one W_Q, W_K, W_V triple, so it can only learn one "style" of relationship between tokens at a time. Multi-head attention runs h independent heads in parallel, each with its own learned projections, on the same Q, K, V inputs:

head_i = Attention(QW_Q^i, KW_K^i, VW_V^i) = softmax((QW_Q^i)(KW_K^i)^T / √d_k) (VW_V^i)

• W_Q^i, W_K^i, W_V^i — head i's own projection matrices, each typically mapping down to a smaller dimension d_k = d_model / h, so total compute across all heads stays comparable to one full-size head.
• Each head computes its own scaled dot-product attention independently over the whole sequence, exactly like single-head attention — just with a different learned projection, so it can end up focusing on a different kind of pattern (e.g. one head tracking subject–verb agreement, another tracking nearby adjectives) purely from what the loss pushes each head's weights toward — nobody assigns heads a "job" explicitly.

Concatenation and final projection

Once all h heads have each produced an output of dimension d_k, they're concatenated side by side back into a d_model-dimensional vector, then passed through one more learned projection W_O:

MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W_O

W_O mixes information across heads back into a single unified representation per token, so the next layer sees one vector per position, not h separate ones.

Why multiple heads help

• Different relationship types at once. One head's dot-product similarity can only capture one "kind" of alignment between tokens per layer; different heads, with different projections, can specialize toward different syntactic/semantic relationships (coreference, local word order, long-range dependency) in the same layer, instead of forcing one attention pattern to do all that work.
• Robustness. If one head learns a noisy or redundant pattern, the others still contribute useful signal — the final W_O projection can down-weight a less useful head rather than the whole layer's representational power resting on a single attention map.
• Same overall compute, more representational power. Since each head works in a smaller d_k = d_model/h dimension, splitting into h heads costs about the same total FLOPs as one full-size head, while giving the model h independent "views" of the sequence instead of one.

Simple shabdo mein: ek hi attention head sirf ek type ka relationship dekh sakta hai; multiple heads parallel mein chalakar model ko sequence ko alag-alag "angles" se dekhne deta hai, bina extra compute cost ke.

Exam tip

When asked "why multiple heads," don't just say "captures different relationships" in the abstract — explicitly mention that each head gets its own W_Q/W_K/W_V, that heads run in reduced dimension d_k = d_model/h so total cost stays flat, and that outputs are concatenated then passed through W_O. That level of detail is what examiners are checking for, not just the intuition.