Multi-Objective Reward and Preference Optimization: Theory and Algorithms

TL;DR

This thesis develops new algorithms and theoretical frameworks for constrained reinforcement learning, preference learning, and large language model alignment, achieving state-of-the-art performance and scalability.

Contribution

It introduces novel algorithms like ACPO, e-COP, warmPref-PS, PSPL, and MOPO, unifying constrained RL and preference-based methods with theoretical guarantees and practical scalability.

Findings

ACPO achieves state-of-the-art empirical performance with theoretical guarantees.

e-COP provides provable performance in episodic constrained RL environments.

MOPO scales to multi-billion-parameter language models for alignment.

Abstract

This thesis develops theoretical frameworks and algorithms that advance constrained reinforcement learning (RL) across control, preference learning, and alignment of large language models. The first contribution addresses constrained Markov Decision Processes (CMDPs) under the average-cost criterion through the Average-Constrained Policy Optimization (ACPO) algorithm. ACPO integrates sensitivity analysis with trust-region updates to ensure stable constraint handling, achieving state-of-the-art empirical performance with theoretical guarantees. Constrained RL is then extended to finite-horizon settings via e-COP, the first policy optimization method for episodic CMDPs. Built on an episodic policy difference lemma, e-COP offers provable performance, simplicity, and scalability in safety-critical environments. The thesis then investigates reinforcement learning from human preferences. warmPref-PS introduces a posterior sampling strategy for linear bandits that integrates offline preference data from heterogeneous raters into online learning. Explicit modeling of rater competence yields substantial regret reduction and more efficient data collection for RLHF. The PSPL algorithm further advances preference-based RL by jointly sampling reward models and transition dynamics from pairwise trajectory comparisons, providing Bayesian simple-regret guarantees and robust empirical identification of optimal policies. The final contribution applies these methods to large-scale model alignment. A multi-objective constrained optimization view yields MOPO, an iterative algorithm with closed-form updates that scales to multi-billion-parameter language models and remains robust across alignment settings. Collectively, the thesis unifies constrained RL across average-cost, episodic, and preference-driven paradigms, delivering theoretical advances and practical tools for safe and aligned decision-making.