CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

Zhuoyi Yang, tengjiayan, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, 杨远明, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, zR, 王维汉, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, Jie Tang

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer: 37 upvotes on Hugging Face Daily Papers, #5 of 11 papers on 2024-08-13. Day-by-day upvote history.

We introduce CogVideoX, a large-scale diffusion transformer model designed for generating videos based on text prompts. To efficently model video data, we propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions. To improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. By employing a progressive training technique, CogVideoX is adept at producing coherent, long-duration videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method. It significantly helps enhance the performance of CogVideoX, improving both generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weights of both the 3D Causal VAE and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.