Uncovering mesa-optimization algorithms in Transformers.

Sun, 01 Jan 2023 00:00:00 +0100

Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, João Sacramento:
Uncovering mesa-optimization algorithms in Transformers. CoRR abs/2309.05858 (2023)