What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao

What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?: 38 upvotes on Hugging Face Daily Papers, #4 of 38 papers on 2026-11-05. Day-by-day upvote history.

Upvotes over time

It had 33 upvotes by its first day, 35 upvotes by the next day, 39 upvotes by day 3, 38 upvotes by day 7, and has 38 now. It ranked #32 of 207 papers in its week and #48 of 340 in its month. Its code repository has 18 GitHub stars.

While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth K and interaction depth T via Pass@(K,T), we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: https://rethinking-agentic-rl.github.io/

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.