CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

TL;DR

CriterAlign is a novel framework that enhances pairwise code preference prediction by using criterion-centric judgments and human-preference-aligned guidance, outperforming traditional monolithic judges.

Contribution

It introduces a criterion-centric approach and HPAG to improve pairwise code evaluation accuracy over existing monolithic methods.

Findings

CriterAlign improves accuracy from 60.4% to 66.3% on BigCodeReward.

Pairwise criterion design significantly contributes to performance gains.

HPAG effectively aligns judgments with human preferences.

Abstract

Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.