Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying

Adithya Renduchintala, Tugrul Konuk, Oleksii Kuchaiev

Tied-Lora: Enhacing parameter efficiency of LoRA with weight tying: 16 upvotes on Hugging Face Daily Papers, #5 of 8 papers on 2023-11-17. Day-by-day upvote history.

We propose Tied-LoRA, a simple paradigm utilizes weight tying and selective training to further increase parameter efficiency of the Low-rank adaptation (LoRA) method. Our investigations include all feasible combinations parameter training/freezing in conjunction with weight tying to identify the optimal balance between performance and the number of trainable parameters. Through experiments covering a variety of tasks and two base language models, we provide analysis revealing trade-offs between efficiency and performance. Our experiments uncovered a particular Tied-LoRA configuration that stands out by demonstrating comparable performance across several tasks while employing only 13~\% percent of parameters utilized by the standard LoRA method.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.