Bootstrapping LLMs via Preference-Based Policy Optimization

TL;DR

This paper introduces a novel preference-based policy optimization framework for bootstrapping large language models, leveraging a min-max game between the policy and reward model, with theoretical guarantees and superior experimental performance.

Contribution

It proposes a new iterative online algorithm for preference-based policy optimization with theoretical regret bounds and improved benchmark results.

Findings

Outperforms existing preference optimization methods.

Provides theoretical guarantees with high-probability regret bounds.

Demonstrates effectiveness across five benchmark datasets.

Abstract

Bootstrapping large language models (LLMs) through preference-based policy optimization offers a promising direction for aligning model behavior with human preferences without relying on extensive manual annotations. In this work, we propose a novel preference-based policy optimization (PbPO) framework that formulates the learning process as a min-max game between the main policy and a reward model (RM). The RM is constrained within a confidence set derived from preference data to ensure reliable exploitation. Our iterative online algorithm actively collects preference data through guided exploration of the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees for our method, establishing high-probability regret bounds for both settings with sequence-level RM and token-level RM, demonstrating its effectiveness in bootstrapping LLMs. Extensive experiments on five benchmarks show that our approach consistently outperforms existing state-of-the-art preference optimization techniques.