Attention computes a weighted average over a set of positions — and a set has no order. Shuffle the words in a sentence before feeding them to a raw self-attention layer, and the output is mathematically identical, permuted the same way. Every scheme on this page exists to inject the one piece of information attention structurally cannot supply on its own: where each token sits.
Self-attention consumes more of a transformer's compute than any other single operation, and its exact mechanics — how heads split the embedding, how masking works, how memory scales — decide almost everything about a model's practical cost, from training time to how long a context window is affordable to serve.
The 2017 paper that introduced the transformer had a blunt thesis: attention was the useful part of the encoder-decoder architecture, so delete everything else. No recurrence, no convolution — just attention and simple feedforward layers, stacked. Removing the sequential dependency of recurrence is what turned scale from a research curiosity into an engineering problem that money and hardware could actually solve.
The architecture that took over NLP turns out to work for images too, once you accept one reframing: an image is a sequence of patches. Once that step is taken, the entire transformer stack from Transformer Architecture transfers to vision essentially unchanged — the same attention, the same feed-forward blocks, the same residual and normalisation pattern.