Not One to Rule Them All: Mining Meaningful Code Review Orders From GitHub

TL;DR

This paper analyzes how developers navigate code review comments on GitHub, revealing diverse meaningful review orders beyond alphabetical, with implications for improving review support especially in larger pull requests.

Contribution

It uncovers various natural review orders used by developers and highlights the need for enhanced tools to support complex review strategies.

Findings

44.6% of reviews follow non-alphabetical order

20.6% follow largest-diff-first order

29% follow a test-first review order

Abstract

Developers use tools such as GitHub pull requests to review code, discuss proposed changes, and request modifications. While changed files are commonly presented in alphabetical order, this does not necessarily coincide with the reviewer's preferred navigation sequence. This study investigates the different navigation orders developers follow while commenting on changes submitted in pull requests. We mined code review comments from 23,241 pull requests in 100 popular Java and Python repositories on GitHub to analyze the order in which the reviewers commented on the submitted changes. Our analysis shows that for 44.6% of pull requests, the reviewers comment in a non-alphabetical order. Among these pull requests, we identified traces of alternative meaningful orders: 20.6% (2,134) followed a largest-diff-first order, 17.6% (1,827) were commented in the order of the files' similarity to the pull request's title and description, and 29% (1,188) of pull requests containing changes to both production and test files adhered to a test-first order. We also observed that the proportion of reviewed files to total submitted files was significantly higher in non-alphabetically ordered reviews, which also received slightly fewer approvals from reviewers, on average. Our findings highlight the need for additional support during code reviews, particularly for larger pull requests, where reviewers are more likely to adopt complex strategies rather than following a single predefined order.