Rethinking Software Misconfigurations in the Real World: An Empirical Study and Literature Analysis

TL;DR

This paper presents an empirical analysis of 772 real-world software misconfigurations, proposes a new root cause classification, and reviews existing literature to identify gaps and trends in misconfiguration troubleshooting.

Contribution

It introduces a novel classification of misconfiguration causes and provides a comprehensive review of research trends and tool availability in the field.

Findings

Research focus has shifted to cloud and advanced applications.

Growth in non-crash misconfiguration studies.

Limited publicly available tools and datasets.

Abstract

Software misconfiguration has consistently been a major reason for software failures. Over the past two decades, much work has been done to detect and diagnose software misconfigurations. However, there is still a gap between real-world misconfigurations and the literature. It is desirable to investigate whether existing taxonomy and tools are applicable for real-world misconfigurations in modern software. In this paper, we conduct an empirical study on 772 real-world misconfiguration issues, based on which we propose a novel classification of the root causes of software misconfigurations, i.e., constraint violation, resource unavailability, component-dependency error, and configuration semantic misinterpretation. Then, we systematically review the literature on misconfiguration troubleshooting to study the trends of research and the practicality of the tools and datasets in this field. We find that the research targets have changed from system and infrastructure software to advanced applications (e.g., cloud service). In the meanwhile, the research on non-crash misconfigurations also has significant growth. Despite the progress, a majority of studies lack reproducibility due to the unavailable tools and evaluation datasets. In total, only ten tools and four datasets are publicly available. We analyze the trends of existing literature on misconfiguration troubleshooting, summarize the challenges that users are faced with, and highlight the suggestions to mitigate and diagnose software misconfigurations. We release the real-world dataset of misconfiguration issues for follow-up research.