An Empirical Validation of Open Source Repository Stability Metrics

TL;DR

This study empirically validates a control theory-based stability metric for open source repositories, demonstrating its practical applicability and suggesting improvements for more accurate project health assessment.

Contribution

First empirical validation of the Composite Stability Index (CSI) for open source repositories, with practical recommendations for data sampling and statistical methods.

Findings

Weekly commit sampling improves stability measurement.

Median-based statistics enhance stability index accuracy.

Data-driven parameters better reflect real project behavior.

Abstract

Over the past few decades, open source software has been continuously integrated into software supply chains worldwide, drastically increasing reliance and dependence. Because of the role this software plays, it is important to understand ways to measure and promote its stability and potential for sustainability. Recent work proposed the use of control theory to understand repository stability and evaluate repositories' ability to return to equilibrium after a disturbance such as the introduction of a new feature request, a spike in bug reports, or even the influx or departure of contributors. This approach leverages commit frequency patterns, issue resolution rate, pull request merge rate, and community activity engagement to provide a Composite Stability Index (CSI). While this framework has theoretical foundations, there is no empirical validation of the CSI in practice. In this paper, we present the first empirical validation of the proposed CSI by experimenting with 100 highly ranked GitHub repositories. Our results suggest that (1) sampling weekly commit frequency pattern instead of daily is a more feasible measure of commit frequency stability across repositories and (2) improved statistical inferences (swapping mean with median), particularly with ascertaining resolution and review times in issues and pull request, improves the overall issue and pull request stability index. Drawing on our empirical dataset, we also derive data-driven half-width parameters that better align stability scores with real project behavior. These findings both confirm the viability of a control-theoretic lens on open-source health and provide concrete, evidence-backed applications for real-world project monitoring tools.