Positional Encodings
Attention computes a weighted average over a set of positions — and a set has no order. Shuffle the words in a sentence before feeding them to a raw self-attention layer, and the output is mathematically identical, permuted the same way. Every scheme on this page exists to inject the one piece of information attention structurally cannot supply on its own: where each token sits.