Transformers learn through gradual rank increase

Enric Boix, Etai Littwin, Emmanuel Abbe, Samy Bengio, Joshua M. Susskind

Transformers learn through gradual rank increase: 10 upvotes on Hugging Face Daily Papers, #9 of 12 papers on 2023-06-13. Day-by-day upvote history.

We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.