Can defects be fixed with weak test suites? An analysis of 50 defects from Defects4J

TL;DR

This study investigates the effectiveness of weak test suites in fixing real-world software defects, finding that up to 84% of 50 analyzed defects could be correctly repaired, highlighting room for improvement in automated program repair.

Contribution

The paper provides an empirical analysis of defect fixability with weak test suites and compares fault localization and patch generation strategies to guide future improvements.

Findings

Up to 84% of defects could be fixed with weak test suites.

Seven fault localization strategies were identified as useful.

Seven patch generation strategies were summarized and compared.

Abstract

Automated program repair techniques, which target to generating correct patches for real world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated program repair techniques can only repair a small portion of defects while producing a lot of incorrect patches. A possible reason for this low performance is that the test suites of real world programs are usually too weak to guarantee the behavior of the program. To understand to what extent defects can be fixed with weak test suites, we analyzed 50 real world defects from Defects4J, in which we found that up to 84% of them could be correctly fixed. This result suggests that there is plenty of space for current automated program repair techniques to improve. Furthermore, we summarized seven fault localization strategies and seven patch generation strategies that were useful in localizing and fixing these defects, and compared those strategies with current repair techniques. The results indicate potential directions to improve automatic program repair in the future research.