Instead of compressing the entire input into one fixed vector and hoping nothing important got lost, attention lets the decoder look back at every encoder position directly, every time it generates a token, and decide for itself which parts of the input actually matter right now. It's a differentiable lookup — a weighted average, where the weights are learned rather than fixed.
The model outputs a probability distribution over the entire vocabulary at every step — something has to turn that distribution into an actual sequence of chosen tokens. That "something" is decoding, and it's a design decision entirely separate from the model itself: the same weights, decoded two different ways, can produce text that reads as either robotic and repetitive or lively and varied.
The hardest part of working with language models isn't building one — it's knowing, with any confidence, whether the new version is actually better. There's no single number that captures "good," and the real skill in evaluation is knowing each imperfect proxy's specific blind spot well enough to know when it's lying to you.
A pretrained model knows an enormous amount about language and often about the world — and none of it about how to behave. It has no notion that a question deserves a direct answer rather than a continuation of similar-looking text scraped from a forum. Fine-tuning is where capability turns into behaviour, and most complaints of "the model can't do X" are actually behaviour problems in disguise.
Every generative model in this section — from a bigram model to GPT — is doing the same thing: predicting the next token given everything before it. That single objective, applied one token at a time, is the entire engine behind modern text generation; everything else in this section is about how to do that prediction better.
Plain RNNs forget almost everything within a few dozen timesteps — the vanishing-gradient product from Recurrent Neural Networks sees to that. In 1997, long before deep learning was mainstream, a fix was published that kept recurrent networks the dominant sequence architecture for another two decades: give the gradient an additive path through time, gated by learned switches deciding what to keep and what to forget.
Full fine-tuning of a seven-billion-parameter model requires storing gradients and Adam's two moment buffers for every single one of those parameters — memory that a single consumer GPU simply doesn't have. Parameter-efficient fine-tuning methods sidestep this by training a tiny fraction of parameters instead, built on a striking empirical observation: the weight updates that fine-tuning actually needs are far lower-rank than the weight matrices themselves.
Attention computes a weighted average over a set of positions — and a set has no order. Shuffle the words in a sentence before feeding them to a raw self-attention layer, and the output is mathematically identical, permuted the same way. Every scheme on this page exists to inject the one piece of information attention structurally cannot supply on its own: where each token sits.
A language model isn't given labelled examples of "good writing" or "correct facts" — it's given unlabelled text and one instruction: predict something that was hidden. Where the knowledge in a modern language model actually comes from traces back to this single trick, repeated over trillions of tokens.
Every network up to this point has processed a single fixed-size input. Language, audio, and time series don't come in fixed sizes — a sentence can be five words or fifty. Recurrent networks were the first architecture built specifically to handle that: reuse the same weights at every timestep, carrying a hidden state forward as a compressed summary of everything seen so far.
Self-attention consumes more of a transformer's compute than any other single operation, and its exact mechanics — how heads split the embedding, how masking works, how memory scales — decide almost everything about a model's practical cost, from training time to how long a context window is affordable to serve.
Translation, summarisation, and question answering all share a structural challenge: the input and output are both sequences, but rarely the same length, and the alignment between them isn't fixed. The encoder-decoder architecture solved this by splitting the problem into two halves — compress the input, then generate the output — and it framed the field's central problem for the next several years, including the one it couldn't quite solve on its own.
Models don't see text — they see a sequence of integers, and every choice about how those integers are assigned decides what the model can and can never learn. A tokeniser that splits "unbelievable" into three pieces lets the model reuse what it learned about "un-" and "-able" elsewhere; a tokeniser that treats it as one opaque unit has no such option.
The 2017 paper that introduced the transformer had a blunt thesis: attention was the useful part of the encoder-decoder architecture, so delete everything else. No recurrence, no convolution — just attention and simple feedforward layers, stacked. Removing the sequential dependency of recurrence is what turned scale from a research curiosity into an engineering problem that money and hardware could actually solve.
One architecture, three ways to cut it — and which cut you choose determines what the resulting model can actually do. The difference between a model that reads and a model that writes turns out to come down to one thing: the shape of the attention mask.
One-hot encoding a vocabulary treats every word as equally different from every other word — "cat" is exactly as far from "dog" as it is from "bicycle." Word embeddings replace that with geometry: words that appear in similar contexts end up close together in a continuous vector space, and "close" starts to mean "similar" in a way a model can actually use.