Interpreting Coding Agents Through Their Behavioral Trajectories
Ibragim Badertdinov, Alexander
Interpreting Coding Agents Through Their Behavioral Trajectories: 10 upvotes on Hugging Face Daily Papers, #16 of 40 papers on 2026-11-16. Day-by-day upvote history.
Upvotes over time
It had 8 upvotes by its first day, 11 upvotes by the next day, 10 upvotes by day 3, 10 upvotes by day 7, and has 10 now. It ranked #48 of 146 papers in its week and #231 of 520 in its month. Its code repository has 1 GitHub stars.
Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Paper page on Hugging Face · arXiv
Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.