How to characterize the health of an Open Source Software project? A snowball literature review of an emerging practice

TL;DR

This paper presents a comprehensive framework of 104 characteristics across 15 themes to assess the health of open source software projects, aiding both research and practical evaluation.

Contribution

It introduces a structured, literature-based framework of OSS health characteristics, covering socio-technical aspects at project and ecosystem levels.

Findings

Final framework includes 104 health characteristics

Characteristics span socio-technical, community, and deliverables

Framework supports OSS health assessment in research and practice

Abstract

Motivation: Society's dependence on Open Source Software (OSS) and the communities that maintain the OSS is ever-growing. So are the potential risks of, e.g., vulnerabilities being introduced in projects not actively maintained. By assessing an OSS project's capability to stay viable and maintained over time without interruption or weakening, i.e., the OSS health, users can consider the risk implied by using the OSS as is, and if necessary, decide whether to help improve the health or choose another option. However, such assessment is complex as OSS health covers a wide range of sub-topics, and existing support is limited. Aim: We aim to create an overview of characteristics that affect the health of an OSS project and enable the assessment thereof. Method: We conduct a snowball literature review based on a start set of 9 papers, and identify 146 relevant papers over two iterations of forward and backward snowballing. Health characteristics are elicited and coded using structured and axial coding into a framework structure. Results: The final framework consists of 104 health characteristics divided among 15 themes. Characteristics address the socio-technical spectrum of the community of actors maintaining the OSS project, the software and other deliverables being maintained, and the orchestration facilitating the maintenance. Characteristics are further divided based on the level of abstraction they address, i.e., the OSS project-level specifically, or the project's overarching ecosystem of related OSS projects. Conclusion: The framework provides an overview of the wide span of health characteristics that may need to be considered when evaluating OSS health and can serve as a foundation both for research and practice.