Reflections on Software Failure Analysis

TL;DR

This paper reviews 52 failure studies in software engineering, highlighting common issues that affect trust and reproducibility, and proposes future directions like causal analysis, standardization, and tool support for defect analysis.

Contribution

It provides a critical reflection on failure study practices, identifies recurring problems, and suggests improvements for future research in software failure analysis.

Findings

Recurring problems hinder trust and replication.

Need for standardization in failure study conduct.

Proposals for causal chain analysis and tool support.

Abstract

Failure studies are important in revealing the root causes, behaviors, and life cycle of defects in software systems. These studies either focus on understanding the characteristics of defects in specific classes of systems or the characteristics of a specific type of defect in the systems it manifests in. Failure studies have influenced various software engineering research directions, especially in the area of software evolution, defect detection, and program repair. In this paper, we reflect on the conduct of failure studies in software engineering. We reviewed a sample of 52 failure study papers. We identified several recurring problems in these studies, some of which hinder the ability of the engineering community to trust or replicate the results. Based on our findings, we suggest future research directions, including identifying and analyzing failure causal chains, standardizing the conduct of failure studies, and tool support for faster defect analysis.