Improving Code Reviewer Recommendation: Accuracy, Latency, Workload, and Bystanders

TL;DR

This paper evaluates new code reviewer recommendation methods at Meta through multiple A/B tests, focusing on accuracy, latency, workload distribution, and the impact of explicit assignment on review speed.

Contribution

It introduces a new low-latency, accurate recommender, and explores workload balancing and the bystander effect in reviewer assignment, with empirical A/B testing at scale.

Findings

New recommender improved accuracy and latency.

Workload balancing had mixed results.

Explicit assignment reduced review time.

Abstract

The code review team at Meta is continuously improving the code review process. To evaluate the new recommenders, we conduct three A/B tests which are a type of randomized controlled experimental trial. Expt 1. We developed a new recommender based on features that had been successfully used in the literature and that could be calculated with low latency. In an A/B test on 82k diffs in Spring of 2022, we found that the new recommender was more accurate and had lower latency. Expt 2. Reviewer workload is not evenly distributed, our goal was to reduce the workload of top reviewers. We then ran an A/B test on 28k diff authors in Winter 2023 on a workload balanced recommender. Our A/B test led to mixed results. Expt 3. We suspected the bystander effect might be slowing down reviews of diffs where only a team was assigned. We conducted an A/B test on 12.5k authors in Spring 2023 and found a large decrease in the amount of time it took for diffs to be reviewed when a recommended individual was explicitly assigned. Our findings also suggest there can be a discrepancy between historical back-testing and A/B test experimental findings.