The Impact of Documentation on Test Engagement in Pull Requests in OSS

TL;DR

This study examines how testing documentation influences contributor testing behavior in open-source projects, finding a positive correlation that suggests documentation may encourage more testing.

Contribution

It introduces the Test Engagement Ratio (TER) and analyzes its relationship with documentation comprehensiveness across 160 OSS repositories.

Findings

Weak but significant positive correlation between documentation and testing engagement (ρ=0.36, p<0.001)

Stronger correlation (ρ=0.44) in repositories with higher pull request activity

TER is moderately correlated (ρ=0.52, p<0.001) with Test Code Ratio

Abstract

Automated testing is crucial for maintaining open-source software quality. However, motivating contributors to include tests for code changes remains a challenge. While existing interventions, such as code coverage metrics and reviewer feedback, are often reactive and applied only after a pull request is opened, this study investigates whether documentation on testing can serve as a proactive measure to encourage testing behavior. In this work, we investigate the relationship between documentation on testing and contributor testing behavior. We introduce the Test Engagement Ratio (TER) to help understand testing frequency. Using data from 160 OSS repositories, we analyze the relationship between documentation comprehensiveness and TER. Our results show a weak but statistically significant positive correlation ($\rho=0.36$, $p<0.001$), which strengthens to a moderate relationship ($\rho=0.44$) in repositories with higher pull request activity. Documentation categories such as How to Run Tests and How to Write Tests show the strongest correlation with testing engagement. Furthermore, TER is found to be moderately correlated ($\rho=0.52$, $p<0.001$) with Test Code Ratio, providing preliminary evidence of its validity. Our findings suggest that documentation on testing may be associated with increased testing engagement. Future work will explore causality, documentation quality at a granular level, and cross-repository exposure effects.