MaLA-500: Massive Language Adaptation of Large Language Models

Peiqin Lin, Shaoxiong, Jörg Tiedemann, Andre Martins, Hinrich Schütze

MaLA-500: Massive Language Adaptation of Large Language Models: 11 upvotes on Hugging Face Daily Papers, #5 of 7 papers on 2024-01-25. Day-by-day upvote history.

Large language models have advanced the state of the art in natural language processing. However, their predominant design for English or a limited set of languages creates a substantial gap in their effectiveness for low-resource languages. To bridge this gap, we introduce MaLA-500, a novel large language model designed to cover an extensive range of 534 languages. To train MaLA-500, we employ vocabulary extension and continued pretraining on LLaMA 2 with Glot500-c. Our experiments on SIB-200 show that MaLA-500 achieves state-of-the-art in-context learning results. We release MaLA-500 at https://huggingface.co/MaLA-LM

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.