DivMerge: A divergence-based model merging method for multi-tasking

Touayouch Brahim, loic fosse, Damnati Géraldine, Lecorvé Gwénolé

DivMerge: A divergence-based model merging method for multi-tasking: 26 upvotes on Hugging Face Daily Papers, #7 of 25 papers on 2025-09-09. Day-by-day upvote history.

Multi-task learning (MTL) is often achieved by merging datasets before fine-tuning, but the growing availability of fine-tuned models has led to new approaches such as model merging via task arithmetic. A major challenge in this setting is task interference, which worsens as the number of tasks increases. We propose a method that merges models trained on different tasks into a single model, maintaining strong performance across all tasks. Our approach leverages Jensen-Shannon divergence to guide the merging process without requiring additional labelled data, and automatically balances task importance. Unlike existing methods, our approach remains robust as the number of tasks grows and consistently outperforms prior work.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.