First Come First Served: The Impact of File Position on Code Review
Enrico Fregnan, Larissa Braz, Marco D'Ambros, G\"ul \c{C}alikli,, Alberto Bacchelli

TL;DR
This study investigates how the position of files in code review tools influences review activity and defect detection, revealing that earlier presented files tend to receive more comments and higher defect detection rates.
Contribution
It provides empirical evidence that file position affects review outcomes and highlights implications for improving code review tool design.
Findings
Earlier files receive more review comments.
File position impacts defect detection odds.
Participants are 64% less likely to find defects in last-position files.
Abstract
The most popular code review tools (e.g., Gerrit and GitHub) present the files to review sorted in alphabetical order. Could this choice or, more generally, the relative position in which a file is presented bias the outcome of code reviews? We investigate this hypothesis by triangulating complementary evidence in a two-step study. First, we observe developers' code review activity. We analyze the review comments pertaining to 219,476 Pull Requests (PRs) from 138 popular Java projects on GitHub. We found files shown earlier in a PR to receive more comments than files shown later, also when controlling for possible confounding factors: e.g., the presence of discussion threads or the lines added in a file. Second, we measure the impact of file position on defect finding in code review. Recruiting 106 participants, we conduct an online controlled experiment in which we measure…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
