The encoder and decoder are lightweight models. The encoder takes in raw input bytes and creates the patch representations that are fed to the global transformer.