Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review

TL;DR

Sphinx is a comprehensive framework that enhances LLM-based pull request review by generating context-rich comments, evaluating review quality with structured benchmarks, and training models with reward optimization to improve accuracy and coverage.

Contribution

The paper introduces Sphinx, a novel unified framework combining data generation, structured evaluation, and reward-based training for improved LLM-driven PR review.

Findings

Models trained with Sphinx achieve up to 40% higher checklist coverage.

Sphinx's evaluation benchmark moves beyond BLEU to assess review quality.

State-of-the-art performance in review completeness and precision.

Abstract

Pull request (PR) review is essential for ensuring software quality, yet automating this task remains challenging due to noisy supervision, limited contextual understanding, and inadequate evaluation metrics. We present Sphinx, a unified framework for LLM-based PR review that addresses these limitations through three key components: (1) a structured data generation pipeline that produces context-rich, semantically grounded review comments by comparing pseudo-modified and merged code; (2) a checklist-based evaluation benchmark that assesses review quality based on structured coverage of actionable verification points, moving beyond surface-level metrics like BLEU; and (3) Checklist Reward Policy Optimization (CRPO), a novel training paradigm that uses rule-based, interpretable rewards to align model behavior with real-world review practices. Extensive experiments show that models trained with Sphinx achieve state-of-the-art performance on review completeness and precision, outperforming both proprietary and open-source baselines by up to 40\% in checklist coverage. Together, Sphinx enables the development of PR review models that are not only fluent but also context-aware, technically precise, and practically deployable in real-world development workflows. The data will be released after review.