ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

TL;DR

ConfPO is a lightweight, model-free method that improves preference alignment in large language models by focusing optimization on critical tokens identified through policy confidence, outperforming prior uniform adjustment methods.

Contribution

It introduces ConfPO, a novel token selection approach based solely on policy confidence, enhancing alignment quality without auxiliary models or extra computation.

Findings

ConfPO outperforms uniform DAAs on benchmark tasks.

It achieves better alignment with zero additional computational cost.

ConfPO reduces overoptimization and reward hacking.

Abstract

We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.