An Exploratory Study of Field Failures

TL;DR

This study analyzes field failures across multiple applications to understand their nature, revealing that a significant portion of faults are inherently difficult to detect before deployment, highlighting the need for runtime solutions.

Contribution

It provides a taxonomy of field failures and discusses the limitations of in-house verification, emphasizing the importance of runtime approaches for certain fault classes.

Findings

70% of faults are intrinsically hard to detect at design time

Analysis of five applications across three ecosystems

Highlights the need for runtime failure detection methods

Abstract

Field failures, that is, failures caused by faults that escape the testing phase leading to failures in the field, are unavoidable. Improving verification and validation activities before deployment can identify and timely remove many but not all faults, and users may still experience a number of annoying problems while using their software systems. This paper investigates the nature of field failures, to understand to what extent further improving in-house verification and validation activities can reduce the number of failures in the field, and frames the need of new approaches that operate in the field. We report the results of the analysis of the bug reports of five applications belonging to three different ecosystems, propose a taxonomy of field failures, and discuss the reasons why failures belonging to the identified classes cannot be detected at design time but shall be addressed at runtime. We observe that many faults (70%) are intrinsically hard to detect at design-time.