Weight subcloning: direct initialization of transformers using larger pretrained ones

Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, Mohammad Rastegari

Weight subcloning: direct initialization of transformers using larger pretrained ones: 18 upvotes on Hugging Face Daily Papers, #7 of 15 papers on 2023-12-18. Day-by-day upvote history.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.