LSTM and GRU
Plain RNNs forget almost everything within a few dozen timesteps — the vanishing-gradient product from Recurrent Neural Networks sees to that. In 1997, long before deep learning was mainstream, a fix was published that kept recurrent networks the dominant sequence architecture for another two decades: give the gradient an additive path through time, gated by learned switches deciding what to keep and what to forget.