Deep Reinforcement Learning from Hierarchical Preference Design

TL;DR

This paper introduces HERON, a hierarchical reward modeling framework that leverages structured preference data to improve reward design, leading to more efficient and robust reinforcement learning agents in complex tasks.

Contribution

The paper proposes a novel hierarchical reward modeling framework, HERON, which exploits structured preference data to simplify reward design and enhance RL performance.

Findings

HERON achieves high performance on various challenging RL tasks.

The framework improves sample efficiency in reinforcement learning.

HERON enhances robustness of learned policies.

Abstract

Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness. Our code is available at \url{https://github.com/abukharin3/HERON}.