SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research

Jinwoo Hwang, Sukmin Cho, Tao Zhang, Soheil Abbasloo, Jongse Park, Peng Cheng

SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research: 12 upvotes on Hugging Face Daily Papers, #13 of 39 papers on 2026-11-10. Day-by-day upvote history.

Upvotes over time

It had 12 upvotes by its first day, 12 upvotes by the next day, 12 upvotes by day 3, 12 upvotes by day 7, and has 12 now. It ranked #50 of 133 papers in its week and #148 of 340 in its month.

LLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.