SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher

Tuan Trung Dao, Thuan Nguyen Hoang, Thành Lê, Duc Vu, Khoi Nguyen, Cuong Pham, ANH TRAN

SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher: 62 upvotes on Hugging Face Daily Papers, #1 of 14 papers on 2024-08-27. Day-by-day upvote history.

In this paper, we aim to enhance the performance of SwiftBrush, a prominent one-step text-to-image diffusion model, to be competitive with its multi-step Stable Diffusion counterpart. Initially, we explore the quality-diversity trade-off between SwiftBrush and SD Turbo: the former excels in image diversity, while the latter excels in image quality. This observation motivates our proposed modifications in the training methodology, including better weight initialization and efficient LoRA training. Moreover, our introduction of a novel clamped CLIP loss enhances image-text alignment and results in improved image quality. Remarkably, by combining the weights of models trained with efficient LoRA and full training, we achieve a new state-of-the-art one-step diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and multi-step Stable Diffusion models. The evaluation code is available at: https://github.com/vinairesearch/swiftbrushv2.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.