Robust Reward Alignment via Hypothesis Space Batch Cutting

TL;DR

This paper introduces a robust reward alignment method for reinforcement learning that uses hypothesis space batched cutting, effectively handling false human preferences and improving robustness over existing techniques.

Contribution

The paper proposes a novel hypothesis space batched cutting approach that enhances robustness to false preferences without explicitly identifying them.

Findings

Achieves comparable or better performance than state-of-the-art in error-free settings.

Significantly outperforms existing methods with high false preference rates.

Provides provable robustness against erroneous human preferences.

Abstract

Reward design in reinforcement learning and optimal control is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in a model predictive control setting across diverse tasks. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences.