What Do Contribution Guidelines Say About Software Testing?
Bruna Falcucci, Felipe Gomide, Andre Hora

TL;DR
This empirical study examines how open-source Python and JavaScript projects communicate testing practices in contribution guidelines, revealing common patterns and gaps in test documentation for contributors.
Contribution
It provides a detailed analysis of testing-related guidance in contribution guidelines across 200 open-source projects, highlighting prevalent practices and areas lacking sufficient information.
Findings
78% of projects include test documentation
Test guidance mainly explains how to run tests (83.5%)
Less than half provide guidance on writing tests (37%)
Abstract
Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78\% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including \texttt{CONTRIBUTING} files (58\%), external documentation (24\%), and \texttt{README} files (8\%). Furthermore, test documentation commonly explains…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Software Testing and Debugging Techniques · Open Source Software Innovations
