Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Hans Zhao, Li Yunshui, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Rosy24, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, cyl, Junyang Lin, shuai bai, Andreas Vlachos, Xu Tan, Zhang, Wen Xiao, Aaron Yee, Tianyu Liu, Baobao Chang

Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey: 53 upvotes on Hugging Face Daily Papers, #3 of 11 papers on 2024-12-30. Day-by-day upvote history. It lost 8 votes when the Hub removed votes in bulk.

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success. As Large Language Models (LLMs) have advanced to unify understanding and generation tasks within the textual modality, recent research has shown that tasks from different modalities can also be effectively encapsulated within the NTP framework, transforming the multimodal information into tokens and predict the next one given the context. This survey introduces a comprehensive taxonomy that unifies both understanding and generation within multimodal learning through the lens of NTP. The proposed taxonomy covers five key aspects: Multimodal tokenization, MMNTP model architectures, unified task representation, datasets \& evaluation, and open challenges. This new taxonomy aims to aid researchers in their exploration of multimodal intelligence. An associated GitHub repository collecting the latest papers and repos is available at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.