Contextual embeddings: same word, different sentences

What's actually being asked

Pass three semantically different sentences through a Transformer encoder, analyze their contextual representations, and explain why the same word ends up with different embeddings in different contexts.

Answer

Setup — three sentences sharing a word

Take the word "bank" appearing in three different sentences:

1. "I deposited money at the bank." (financial institution)
2. "We sat on the river bank." (edge of a river)
3. "The pilot had to bank the plane sharply." (to tilt/turn, verb)

Feed each sentence independently through the same Transformer encoder.

What happens inside the encoder

Each token starts from a static embedding (the same vector for "bank" regardless of sentence — from the embedding lookup table, plus positional encoding). What turns it into something sentence-specific is self-attention:

Attention(Q,K,V) = softmax(QK^T/√d_k) V, with Q = XW_Q, K = XW_K, V = XW_V

In sentence 1, "bank"'s query vector computes attention scores against every other token's key — "deposited" and "money" score highly (their learned key projections make them a strong match for "bank"'s query here), so "bank"'s output representation becomes a weighted blend pulling heavily from "deposited" and "money"'s value vectors.

In sentence 2, "bank"'s query instead scores highly against "river" and "sat" — its output representation pulls from a completely different set of value vectors, landing in a different region of representation space (closer to other "geography/edge" concepts).

In sentence 3, "bank" is used as a verb, next to "pilot" and "plane" — attention pulls in those tokens' values, and because position/role also differs (verb vs. noun position), the resulting vector again lands somewhere distinct, likely nearer to other motion/action-verb representations.

This repeats through every one of the encoder's N stacked layers, each layer re-mixing every token's representation based on the current (already-contextualized) representations of all other tokens — so the contextual blending compounds across depth, not just in one shot.

Why the same word gets different embeddings

• The input embedding for "bank" (the lookup-table vector before any attention) is identical across all three sentences — this part is static, just like a classic word2vec/GloVe vector.
• But self-attention makes every layer's output representation for a token a weighted average of the entire sentence's value vectors, with the weights depending on that specific sentence's Q·K similarities. Since the surrounding tokens (and thus the keys/values being attended to) differ in each sentence, the resulting blend differs — "bank" ends up at a different point in representation space each time.
• Formally: output("bank" in sentence_k) = Σ_j softmax(Q_bank · K_j / √d_k)_j · V_j, and the set {K_j, V_j} is a different set of tokens (financial words vs. geography words vs. motion words) in each sentence, so the weighted sum necessarily lands in a different place.

This is precisely what separates contextual embeddings (BERT, GPT, any Transformer encoder output) from classic static embeddings (word2vec, GloVe) — a static embedding gives "bank" exactly one vector, period; a contextual embedding gives it a different vector every time it appears in a different context, because attention re-derives that vector from the specific sentence it's in.

Exam tip

If asked to "analyze," don't stop at "different sentences give different vectors" — explicitly connect it to the attention formula and explain which other tokens pull the representation in which direction for each sentence. That causal explanation, via Q·K similarity and the weighted V sum, is what earns full marks over a one-line static-vs-contextual claim.