An Empirical Study on the Amount of Changes Required for Merge Request Acceptance

TL;DR

This study analyzes the effort involved in code review for GitLab Merge Requests, revealing that a significant portion requires extensive changes, and demonstrates that machine learning can effectively predict review effort based on various project and code features.

Contribution

It introduces a comprehensive measurement of code review effort based on code modifications and develops an interpretable machine learning model to predict review effort using diverse features.

Findings

Up to 71% of MRs require post-submission adjustments.

28% of adjustments involve over 200 lines of code.

Machine learning models achieve high accuracy (AUC 0.84-0.88) in predicting review effort.

Abstract

Code review (CR) is essential to software development, helping ensure that new code is properly integrated. However, the CR process often involves significant effort, including code adjustments, responses to reviewers, and continued implementation. While past studies have examined CR delays and iteration counts, few have investigated the effort based on the volume of code changes required, especially in the context of GitLab Merge Requests (MRs), which remains underexplored. In this paper, we define and measure CR effort as the amount of code modified after submission, using a dataset of over 23,600 MRs from four GitLab projects. We find that up to 71% of MRs require adjustments after submission, and 28% of these involve changes to more than 200 lines of code. Surprisingly, this effort is not correlated with review time or the number of participants. To better understand and predict CR effort, we train an interpretable machine learning model using metrics across multiple dimensions: text features, code complexity, developer experience, review history, and branching. Our model achieves strong performance (AUC 0.84-0.88) and reveals that complexity, experience, and text features are key predictors. Historical project characteristics also influence current review effort. Our findings highlight the feasibility of using machine learning to explain and anticipate the effort needed to integrate code changes during review.