ReplaceMe: Network Simplification via Layer Pruning and Linear Transformations

Dmitry Shopkhoev, Ammar Ali, Magauiya Zhussip, Valentin Malykh, Stamatis Lefkimmiatis, Nikos Komodakis, Sergey Zagoruyko

ReplaceMe: Network Simplification via Layer Pruning and Linear Transformations: 19 upvotes on Hugging Face Daily Papers, #8 of 22 papers on 2025-05-06. Day-by-day upvote history. It lost 7 votes when the Hub removed votes in bulk.

We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast to conventional pruning approaches that require additional training or fine-tuning, our approach requires only a small calibration dataset that is used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seamlessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-free approaches and remains highly competitive with state-of-the-art pruning methods that involve extensive retraining/fine-tuning and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25\% pruning while retaining approximately 90\% of the original model's performance on open benchmarks - without any training or healing steps, resulting in minimal computational overhead. We provide an open-source library implementing ReplaceMe alongside several state-of-the-art depth pruning techniques, available at https://github.com/mts-ai/ReplaceMe

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.