On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer

Zhenyi Shen, Junrulu, Wei Liu, Jiangnan Ye, Jiazheng Li, DI YIN, Xing Sun, Tameem Adel, Yali Du, Lin Gui, Yulan He

On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer: 16 upvotes on Hugging Face Daily Papers, #13 of 32 papers on 2026-11-06. Day-by-day upvote history.

Upvotes over time

It had 12 upvotes by its first day, 14 upvotes by the next day, 15 upvotes by day 3, 16 upvotes by day 7, and has 16 now. It ranked #82 of 207 papers in its week and #122 of 340 in its month. Its code repository has 10 GitHub stars.

Prefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.