Assessing the Impact of File Ordering Strategies on Code Review Process

TL;DR

This study investigates how file ordering strategies in code review affect review quality, confirming that alternative orderings like code diff can improve detection of problematic files compared to standard alphabetical sorting.

Contribution

It verifies prior findings on a large industrial project and introduces an alternative file ordering strategy based on code diff, showing its effectiveness in enhancing review focus.

Findings

Files higher in review order receive more comments.

Ordering by code diff improves identification of problematic files.

Alphabetical order skews review efforts towards certain files.

Abstract

Popular modern code review tools (e.g. Gerrit and GitHub) sort files in a code review in alphabetical order. A prior study (on open-source projects) shows that the changed files' positions in the code review affect the review process. Their results show that files placed lower in the order have less chance of receiving reviewing efforts than the other files. Hence, there is a higher chance of missing defects in these files. This paper explores the impact of file order in the code review of the well-known industrial project IntelliJ IDEA. First, we verify the results of the prior study on a big proprietary software project. Then, we explore an alternative to the default Alphabetical order: ordering changed files according to their code diff. Our results confirm the observations of the previous study. We discover that reviewers leave more comments on the files shown higher in the code review. Moreover, these results show that, even with the data skewed toward Alphabetical order, ordering changed files according to their code diff performs better than standard Alphabetical order regarding placing problematic files, which needs more reviewing effort, in the code review. These results confirm that exploring various ordering strategies for code review needs more exploration.