A Note About: Critical Review of BugSwarm for Fault Localization and Program Repair

TL;DR

This paper defends the BugSwarm dataset against a critical review, replicates the study, corrects misconceptions, and discusses its proper use for fault localization and program repair research.

Contribution

It provides a replication of the critical review, corrects inaccuracies about BugSwarm, and offers guidance on its appropriate application in software engineering research.

Findings

Several claims in the critical review are incorrect.

BugSwarm's dataset is valuable for fault localization and repair.

The paper clarifies misconceptions and suggests proper usage of BugSwarm.

Abstract

Datasets play an important role in the advancement of software tools and facilitate their evaluation. BugSwarm is an infrastructure to automatically create a large dataset of real-world reproducible failures and fixes. In this paper, we respond to Durieux and Abreu's critical review of the BugSwarm dataset, referred to in this paper as CriticalReview. We replicate CriticalReview's study and find several incorrect claims and assumptions about the BugSwarm dataset. We discuss these incorrect claims and other contributions listed by CriticalReview. Finally, we discuss general misconceptions about BugSwarm, and our vision for the use of the infrastructure and dataset.