Applicability of Software Reliability Growth Models to Open Source Software

TL;DR

This study evaluates the applicability of various Software Reliability Growth Models to open-source projects, analyzing 88 projects to understand their performance and limitations across different domains and project segments.

Contribution

It provides a comprehensive experimental analysis of nine SRGMs on a large OSS dataset, using automation for reproducibility and highlighting domain-specific model performance.

Findings

SRGMs show good applicability to OSS.

Model performance varies across releases and domains.

Generalizing SRGM effectiveness remains challenging.

Abstract

Software Reliability Growth Models (SRGMs) are based on underlying assumptions which make them typically more suited for quality evaluation of closed-source projects and their development lifecycles. Their usage in open-source software (OSS) projects is a subject of debate. Although the studies investigating the SRGMs applicability in OSS context do exist, they are limited by the number of models and projects considered which might lead to inconclusive results. In this paper, we present an experimental study of SRGMs applicability to a total of 88 OSS projects, comparing nine SRGMs, looking at the stability of the best models on the whole projects, on releases, on different domains, and according to different projects' attributes. With the aid of the STRAIT tool, we automated repository mining, data processing, and SRGM analysis for better reproducibility. Overall, we found good applicability of SRGMs to OSS, but with different performance when segmenting the dataset into releases and domains, highlighting the difficulty in generalizing the findings and in the search for one-fits-all models.