Law of Vision Representation in MLLMs

Shijia Yang, Bohan Zhai, Quanzeng You, Yuan JianBo, Hongxia Yang, Chenfeng Xu

Law of Vision Representation in MLLMs: 95 upvotes on Hugging Face Daily Papers, #1 of 11 papers on 2024-08-30. Day-by-day upvote history.

We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.