Offline Constrained RLHF with Multiple Preference Oracles

TL;DR

This paper develops a method for offline constrained reinforcement learning from human preferences, ensuring safety and fairness constraints while maximizing utility, with theoretical guarantees and extensions to multiple constraints.

Contribution

It introduces a dual-only algorithm with finite-sample guarantees for offline constrained preference learning, handling multiple constraints and f-divergence regularization.

Findings

Proposed a dual-only algorithm with high-probability constraint satisfaction.

Provided the first finite-sample performance guarantees for offline constrained preference learning.

Extended analysis to multiple constraints and general f-divergence regularization.

Abstract

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.