Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution

TL;DR

PEPO is a new algorithm that mitigates over-optimization in preference learning by using an ensemble approach to promote conservative policy updates without needing data distribution knowledge.

Contribution

The paper proposes PEPO, a single-step ensemble method that avoids over-optimization in preference learning without requiring explicit reward models or distribution knowledge.

Findings

PEPO guarantees sample complexity depending only on a single-policy concentrability coefficient.

PEPO outperforms traditional DPO in avoiding over-optimization issues.

Theoretical results are supported by practical experiments.

Abstract

We introduce PEPO (Pessimistic Ensemble based Preference Optimization), a single-step Direct Preference Optimization (DPO)-like algorithm to mitigate the well-known over-optimization issue in preference learning without requiring the knowledge of the data-generating distribution or learning an explicit reward model. PEPO achieves pessimism via an ensemble of preference-optimized policies trained on disjoint data subsets and then aggregates them through a worst case construction that favors the agreement across models. In the tabular setting, PEPO achieves sample complexity guarantees depending only on a single-policy concentrability coefficient, thus avoiding the all-policy concentrability which affects the guarantees of algorithms prone to over-optimization, such as DPO. The theoretical findings are corroborated by a convincing practical performance, while retaining the simplicity and the practicality of DPO-style training.