What Do Contribution Guidelines Say About Software Testing?

TL;DR

This empirical study examines how open-source Python and JavaScript projects communicate testing practices in contribution guidelines, revealing common patterns and gaps in test documentation for contributors.

Contribution

It provides a detailed analysis of testing-related guidance in contribution guidelines across 200 open-source projects, highlighting prevalent practices and areas lacking sufficient information.

Findings

78% of projects include test documentation

Test guidance mainly explains how to run tests (83.5%)

Less than half provide guidance on writing tests (37%)

Abstract

Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78\% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including \texttt{CONTRIBUTING} files (58\%), external documentation (24\%), and \texttt{README} files (8\%). Furthermore, test documentation commonly explains how to run tests (83.5\%), but less often provides guidance on how to write tests (37\%). It frequently covers unit tests (71\%), but rarely addresses integration (20.5\%) and end-to-end tests (15.5\%). Other key testing aspects are also less frequently discussed: test coverage (25.5\%) and mocking (9.5\%). We conclude by discussing implications and future research.