Continuous-Utility Direct Preference Optimization

TL;DR

CU-DPO introduces a continuous scoring framework for model preference optimization, improving reasoning accuracy and efficiency over binary preference methods by capturing fine-grained reasoning quality.

Contribution

It proposes a novel continuous utility framework and a two-stage training pipeline that significantly enhances reasoning performance and sample efficiency.

Findings

Strategy selection accuracy improved from 35-46% to 68-78%.

Downstream reasoning gains of up to 6.6 points on benchmarks.

Effective transfer to out-of-distribution tasks.

Abstract

Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce Continuous Utility Direct Preference Optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences, and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage training pipeline: (i) strategy selection, which optimizes the model to choose the best strategy for a given problem via best-vs-all comparisons, and (ii) execution refinement, which trains the model to correctly execute the selected strategy using margin-stratified pairs. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46 percent to 68-78 percent across seven base models, yielding consistent downstream reasoning gains of up to 6.6 points on in-distribution datasets with effective transfer to out-of-distribution tasks.