Individual context-free online community health indicators fail to identify open source software sustainability

TL;DR

This study shows that individual context-free online community health indicators are insufficient for assessing open source software sustainability, emphasizing the importance of contextualized, project-specific evaluation methods.

Contribution

The paper demonstrates that context-free metrics are inadequate for predicting open source project longevity and advocates for combined qualitative and quantitative, project-specific assessments.

Findings

Context-free metrics vary greatly across projects

Similar indicators can have different meanings depending on context

Single-project assessments are more reliable for sustainability evaluation

Abstract

The global value of open source software is estimated to be in the billions or trillions worldwide1, but despite this, it is often under-resourced and subject to high-impact security vulnerabilities and stability failures2,3. In order to investigate factors contributing to open source community longevity, we monitored thirty-eight open source projects over the period of a year, focusing primarily, but not exclusively, on open science-related online code-oriented communities. We measured performance indicators, using both subjective and qualitative measures (participant surveys), as well as using computational scripts to retrieve and analyse indicators associated with these projects' online source control codebases. None of the projects were abandoned during this period, and only one project entered a planned shutdown. Project ages spanned from under one year to over forty years old at the start of the study, and results were highly heterogeneous, showing little commonality across documentation, mean response times for issues and code contributions, and available funding/staffing resources. Whilst source code-based indicators were able to offer some insights into project activity, we observed that similar indicators across different projects often had very different meanings when context was taken into account. We conclude that the individual context-free metrics we studied were not sufficient or essential for project longevity and sustainability, and might even become detrimental if used to support high-stakes decision making. When attempting to understand an online open community's longer-term sustainability, we recommend that researchers avoid cross-project quantitative comparisons, and advise instead that they use single-project-level assessments which combine quantitative measures with contextualising qualitative data.