Investigating Test Overfitting on SWE-bench

TL;DR

This paper empirically investigates test overfitting in issue resolution systems that use auto-generated tests, highlighting potential pitfalls of over-reliance on such tests.

Contribution

First empirical study analyzing test overfitting in issue resolution systems that generate and refine tests automatically.

Findings

Identifies the prevalence of test overfitting in automated issue resolution.

Highlights risks of relying on imperfect auto-generated tests.

Provides insights into the limitations of current test-based issue resolution methods.

Abstract

Tests can be useful towards resolving issues on code repositories. However, relying too much on tests for issue resolution can lead to code that technically passes observed tests but actually misses important cases or even breaks functionality. This problem, called test overfitting, is exacerbated by the fact that issues usually lack readily executable tests. Instead, several issue resolution systems use tests auto-generated from issues, which may be imperfect. Some systems even iteratively refine code and tests jointly. This paper presents the first empirical study of test overfitting in this setting.