RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

TL;DR

This paper introduces RA-PbRL, a new risk-aware preference-based reinforcement learning algorithm that optimizes risk-sensitive objectives, with theoretical guarantees and empirical validation, addressing safety-critical applications.

Contribution

We propose RA-PbRL, the first algorithm to optimize nested and static risk-aware objectives in PbRL, with proven sublinear regret bounds and empirical performance evaluation.

Findings

RA-PbRL achieves sublinear regret bounds.

Empirical results support the effectiveness of risk-aware objectives.

Theoretical analysis confirms the algorithm's efficiency.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has recently surged in popularity, particularly for aligning large language models and other AI systems with human intentions. At its core, RLHF can be viewed as a specialized instance of Preference-based Reinforcement Learning (PbRL), where the preferences specifically originate from human judgments rather than arbitrary evaluators. Despite this connection, most existing approaches in both RLHF and PbRL primarily focus on optimizing a mean reward objective, neglecting scenarios that necessitate risk-awareness, such as AI safety, healthcare, and autonomous driving. These scenarios often operate under a one-episode-reward setting, which makes conventional risk-sensitive objectives inapplicable. To address this, we explore and prove the applicability of two risk-aware objectives to PbRL : nested and static quantile risk objectives. We also introduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize both nested and static objectives. Additionally, we provide a theoretical analysis of the regret upper bounds, demonstrating that they are sublinear with respect to the number of episodes, and present empirical results to support our findings. Our code is available in https://github.com/aguilarjose11/PbRLNeurips.