Self-Attention in Depth
Self-attention consumes more of a transformer's compute than any other single operation, and its exact mechanics — how heads split the embedding, how masking works, how memory scales — decide almost everything about a model's practical cost, from training time to how long a context window is affordable to serve.