Early neural networks for natural language problems used an encoder RNN (recurrent neural network). ... transformers manage context from past and future input using an encoder-decoder structure.