Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages: 22 upvotes on Hugging Face Daily Papers, #9 of 39 papers on 2026-11-10. Day-by-day upvote history.

Upvotes over time

It had 19 upvotes by its first day, 21 upvotes by the next day, 22 upvotes by day 3, 22 upvotes by day 7, and has 22 now. It ranked #35 of 133 papers in its week and #96 of 340 in its month. Its code repository has 3 GitHub stars.

Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver--judge--repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluation tooling, and agent trajectories, together with an additional set of easier tasks to support evaluation of smaller models.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.