BitsAI-CR: Automated Code Review via LLM in Practice

TL;DR

BitsAI-CR is a practical framework that leverages large language models for automated code review, combining rule-based detection and verification to improve precision and scalability in industrial settings.

Contribution

The paper introduces a two-stage LLM-based code review system with a feedback-driven performance improvement mechanism and a new Outdated Rate metric for real-world adoption measurement.

Findings

Achieves 75% precision in review comment generation.

Maintains an Outdated Rate of 26.7% for Go language reviews.

Successfully deployed at ByteDance with over 12,000 weekly active users.

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.