UT5: Pretraining Non autoregressive T5 with unrolled denoising

Mahmoud G. Salem, Jiayu Ye, Chu-Cheng Lin, Frederick Liu

UT5: Pretraining Non autoregressive T5 with unrolled denoising: 8 upvotes on Hugging Face Daily Papers, #9 of 13 papers on 2023-11-16. Day-by-day upvote history.

Recent advances in Transformer-based Large Language Models have made great strides in natural language generation. However, to decode K tokens, an autoregressive model needs K sequential forward passes, which may be a performance bottleneck for large language models. Many non-autoregressive (NAR) research are aiming to address this sequentiality bottleneck, albeit many have focused on a dedicated architecture in supervised benchmarks. In this work, we studied unsupervised pretraining for non auto-regressive T5 models via unrolled denoising and shown its SoTA results in downstream generation tasks such as SQuAD question generation and XSum.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.