AstroLLaMA: Towards Specialized Foundation Models in Astronomy

Josh Nguyen, Yuan-Sen Ting, Ioana Ciuca , Charlie O'Neill, Ze-Chang Sun, Maya, Sandor Kruk, Ernest Perkowski, Jack Miller, Jason Jingshi Li, Josh Peek, Kartheik Iyer, Tomasz Różański, Pranav Khetarpal, Sharaf Zaman, David Brodrick, Sergio José Rodríguez Méndez, Thang Bui, Alyssa Goodman, Alberto Accomazzi, JPN, Jesse Cranney, Kevin Schawinski, UniverseTBD

AstroLLaMA: Towards Specialized Foundation Models in Astronomy: 18 upvotes on Hugging Face Daily Papers, #5 of 9 papers on 2023-09-13. Day-by-day upvote history.

Large language models excel in many human-language tasks but often falter in highly specialized domains like scholarly astronomy. To bridge this gap, we introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2, showing marked domain adaptation. Our model generates more insightful and scientifically relevant text completions and embedding extraction than state-of-the-arts foundation models despite having significantly fewer parameters. AstroLLaMA serves as a robust, domain-specific model with broad fine-tuning potential. Its public release aims to spur astronomy-focused research, including automatic paper summarization and conversational agent development.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.