SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

Peng Gao, Renrui, Dongyang Liu (Chris Liu), Longtian Qiu, Siyuan, Weifeng Lin, steve z, Shijie Geng, Ziyi Lin, Peng Jin, KAIPENG ZHANG, SII - Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng LI, Yu Qiao

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models: 17 upvotes on Hugging Face Daily Papers, #8 of 15 papers on 2024-02-09. Day-by-day upvote history.

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.