Using a Balanced Scorecard to Identify Opportunities to Improve Code Review Effectiveness: An Industrial Experience Report

TL;DR

This paper presents a Balanced Scorecard-based approach to monitor and improve code review effectiveness in an industrial setting, achieving significant accuracy improvements over prior models and human evaluators.

Contribution

Developed a production-ready, automated monitoring tool using the Balanced Scorecard strategy to evaluate and enhance code review effectiveness in a real-world industrial environment.

Findings

Automated model improved accuracy by 7.88% over prior models.

Model outperformed human evaluators by 25.32% in accuracy.

Developers and managers found the tool useful for decision-making.

Abstract

Peer code review is a widely adopted software engineering practice to ensure code quality and ensure software reliability in both the commercial and open-source software projects. Due to the large effort overhead associated with practicing code reviews, project managers often wonder, if their code reviews are effective and if there are improvement opportunities in that respect. Since project managers at Samsung Research Bangladesh (SRBD) were also intrigued by these questions, this research developed, deployed, and evaluated a production-ready solution using the Balanced SCorecard (BSC) strategy that SRBD managers can use in their day-to-day management to monitor individual developer's, a particular project's or the entire organization's code review effectiveness. Following the four-step framework of the BSC strategy, we: 1) defined the operation goals of this research, 2) defined a set of metrics to measure the effectiveness of code reviews, 3) developed an automated mechanism to measure those metrics, and 4) developed and evaluated a monitoring application to inform the key stakeholders. Our automated model to identify useful code reviews achieves 7.88% and 14.39% improvement in terms of accuracy and minority class F1 score respectively over the models proposed in prior studies. It also outperforms human evaluators from SRBD, that the model replaces, by a margin of 25.32% and 23.84% respectively in terms of accuracy and minority class F1 score. In our post-deployment survey, SRBD developers and managers indicated that they found our solution as useful and it provided them with important insights to help their decision makings.