LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

siqi zhu, Suozhi Huang, Yi Zhao, Jiaxuan You

LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers: 4 upvotes on Hugging Face Daily Papers, #28 of 42 papers on 2026-11-17. Day-by-day upvote history.

Upvotes over time

It had 4 upvotes by its first day, 4 upvotes by the next day, 4 upvotes by day 3, 4 upvotes by day 7, and has 4 now. It ranked #84 of 146 papers in its week and #361 of 520 in its month.

Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85times on Ouro-2.6B and 5.13times on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by \(40\%\) on Ouro-2.6B and \(25\%\) on Huginn-3.5B.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.