Tell me why: Training preferences-based RL with human preferences and step-level explanations

TL;DR

This paper introduces a preference-based reinforcement learning method that incorporates human-provided explanations for preferences, enhancing learning efficiency by allowing more expressive feedback over individual trajectory steps.

Contribution

It presents a novel approach enabling humans to give preference feedback along with step-level explanations, improving the learning process in preference-based RL.

Findings

Explanations improve learning speed.

Step-level feedback enhances preference accuracy.

Method outperforms traditional preference-based RL in simulations.

Abstract

Human-in-the-loop reinforcement learning allows the training of agents through various interfaces, even for non-expert humans. Recently, preference-based methods (PbRL), where the human has to give his preference over two trajectories, increased in popularity since they allow training in domains where more direct feedback is hard to formulate. However, the current PBRL methods have limitations and do not provide humans with an expressive interface for giving feedback. With this work, we propose a new preference-based learning method that provides humans with a more expressive interface to provide their preference over trajectories and a factual explanation (or annotation of why they have this preference). These explanations allow the human to explain what parts of the trajectory are most relevant for the preference. We allow the expression of the explanations over individual trajectory steps. We evaluate our method in various simulations using a simulated human oracle (with realistic restrictions), and our results show that our extended feedback can improve the speed of learning.