Probability-Consistent Preference Optimization for Enhanced LLM Reasoning

TL;DR

This paper introduces PCPO, a new preference optimization framework for large language models that considers both answer correctness and internal token probability consistency, leading to improved reasoning performance.

Contribution

The paper proposes a novel dual-metric framework, PCPO, which incorporates internal probability consistency into preference optimization for LLMs, enhancing reasoning capabilities.

Findings

PCPO outperforms outcome-only methods across various benchmarks.

Incorporating token-level probability consistency improves LLM reasoning.

Extensive experiments validate the effectiveness of PCPO.

Abstract

Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO.