3 docs tagged with "attention"

Instead of compressing the entire input into one fixed vector and hoping nothing important got lost, attention lets the decoder look back at every encoder position directly, every time it generates a token, and decide for itself which parts of the input actually matter right now. It's a differentiable lookup — a weighted average, where the weights are learned rather than fixed.

This page has not been written yet.

Self-attention consumes more of a transformer's compute than any other single operation, and its exact mechanics — how heads split the embedding, how masking works, how memory scales — decide almost everything about a model's practical cost, from training time to how long a context window is affordable to serve.