Symbol Guided Hindsight Priors for Reward Learning from Human Preferences

TL;DR

This paper introduces the PRIOR framework, which uses structured priors and abstract state spaces to significantly reduce feedback needs and enhance reward recovery in preference-based reinforcement learning.

Contribution

The paper proposes a novel PRIOR framework that incorporates reward and feedback priors, reducing feedback requirements and improving reward learning in PbRL.

Findings

Reduces feedback needed by half for reward recovery.

Improves reward learning accuracy and agent performance.

Using abstract state spaces further enhances reward inference.

Abstract

Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amount of feedback required by half and improves overall reward recovery. Additionally, we demonstrate that using an abstract state space for the computation of the priors further improves the reward learning and the agent's performance.