Test Code Review in the Era of GitHub Actions: A Replication Study

TL;DR

This study investigates how the adoption of GitHub pull requests and Actions affects test code review practices, revealing a decline in test discussions and emphasizing the need for improved review strategies to maintain software quality.

Contribution

It replicates and extends prior research by analyzing the impact of GHA and PR models on test code review across multiple open-source projects.

Findings

GitHub PRs promote more balanced discussions between test and production code than Gerrit.

GHA adoption leads to a significant decrease in test code review activity.

Test review discussions are nearly absent in GHA-enabled PRs, raising concerns about software quality.

Abstract

Test code is indispensable in software development, ensuring the correctness of production code and supporting maintainability. Nonetheless, errors or omissions in the test code can conceal production defects. While code review is widely adopted to assess code quality and correctness, little research has examined how test code is reviewed. Spadini et al.'s research on Gerrit (a pre-commit review model) found that test code receives significantly less discussion than production code. However, the most popular review model is currently based on pull requests (PRs), in which contributors propose changes for discussion and approval, a more negotiable and flexible model compared to Gerrit. Furthermore, GitHub Actions (GHA) has become widely used to automate pre-checks and testing, potentially impacting review practices. This leads us to explore whether Spadini et al.'s findings still hold for the PR model in the era of GHA? Our work replicates and extends their work. We focus on GitHub PRs and analyze six open-source projects. We investigate the impact of the PR model and GHA on test code review. Our results show that GitHub's PR model fosters more balanced discussions between test and production files than Gerrit, albeit with lower overall comment density. However, despite cross-project heterogeneity, GHA adoption triggered a sharp pivot toward production code. Post-GHA, for PRs involving tests, both review probability and comment density reached a median of zero. These findings reveal how evolving continuous integration pipelines can marginalize test code review. The observed decline in test-centric discussion under GHA warrants concern regarding long-term software quality. Our work also presents recommendations for stakeholders involved in the software development life cycle.