2 docs tagged with "sequences"

Plain RNNs forget almost everything within a few dozen timesteps — the vanishing-gradient product from Recurrent Neural Networks sees to that. In 1997, long before deep learning was mainstream, a fix was published that kept recurrent networks the dominant sequence architecture for another two decades: give the gradient an additive path through time, gated by learned switches deciding what to keep and what to forget.

Every network up to this point has processed a single fixed-size input. Language, audio, and time series don't come in fixed sizes — a sentence can be five words or fifty. Recurrent networks were the first architecture built specifically to handle that: reuse the same weights at every timestep, carrying a hidden state forward as a compressed summary of everything seen so far.