LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks

Every benchmark result for an LLM carries an unstated assumption: that the model has not already seen the answers. Nobody had measured whether that holds for the benchmarks software engineering research actually uses.

We quantify overlap between LLM pre-training corpora and 83 SE benchmarks. Average leakage turns out to be low — about 4.8% for Python, 2.8% for Java, and 0.7% for C/C++ benchmarks — but it is distributed very unevenly, and the few heavily contaminated benchmarks are enough to distort conclusions drawn from them. We release LessLeak-Bench, a leakage-filtered version of these benchmarks, so future comparisons start from clean data.