Using a Sledgehammer to Crack a Nut? Revisiting Automated Compiler Fault Isolation

TL;DR

This paper compares traditional BIC-based bug localization with SBFL techniques in compiler fault isolation, finding that BIC-based methods perform comparably or better in practical scenarios, especially on key ranking metrics.

Contribution

It provides the first direct comparison of BIC-based strategies with SBFL techniques for compiler fault localization, highlighting the practical effectiveness of BIC-based approaches.

Findings

BIC-based strategy performs comparably or better than SBFL techniques.

BIC-based approach is effective in real-world compiler debugging.

Future research should consider BIC-based methods as a baseline.

Abstract

Background: Compilers are fundamental to software development, translating high-level source code into executable software systems. Faults in compilers can have severe consequences and thus effective localization and resolution of compiler bugs are crucial. Problem: In practice, developers often examine version history to identify and investigate bug-inducing commit (BIC) for fixing bugs. However, while numerous sophisticated Spectrum-Based Fault Localization (SBFL) techniques have been proposed for compiler fault isolation, their effectiveness has not been evaluated against the BIC-based strategies widely adopted in practice. Objective: This study aims to bridge this gap by directly comparing a BIC-based strategy, Basic, with representative SBFL techniques in the context of compiler fault localization. The BIC-based strategy closely aligns with common developer practices, as it directly identifies the BIC and treats the files modified in that commit as faulty candidates. Method: The Basic identifies the most recent good release and earliest bad release, and then employs a binary search to pinpoint the bug-inducing commit. All files modified in the identified commit are flagged as potentially faulty. We rigorously compare Basic against SBFL-based techniques using a benchmark consisting of 60 GCC bugs and 60 LLVM bugs. Result: Our analysis reveals that Basic performs comparably to, and in many cases outperforms, state-of-the-art SBFL-based techniques, particularly on the critical Top-1 and Top-5 ranking metrics. Conclusion: This study provides new insights into the practical effectiveness of SBFL-based techniques in real-world compiler debugging scenarios. We recommend that future research adopt Basic as a baseline when developing and evaluating new compiler fault isolation methods.