The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

Jonathan katzy, Razvan Popescu, Arie van Deursen, Mali (Maliheh) Izadi

The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models: 10 upvotes on Hugging Face Daily Papers, #10 of 12 papers on 2025-01-17. Day-by-day upvote history.

The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.