Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning

TL;DR

This paper introduces Sub-optimal Data Pre-training (SDP), a method that uses reward-free, low-quality data to pre-train reward models, significantly reducing human interaction needs in human-in-the-loop reinforcement learning.

Contribution

The paper proposes SDP, a novel pre-training approach leveraging sub-optimal data to enhance reward model learning and improve feedback efficiency in human-in-the-loop RL.

Findings

SDP improves reward model training without human labels.

SDP enhances RL performance across robotic tasks.

SDP reduces human interaction requirements.

Abstract

To create useful reinforcement learning (RL) agents, step zero is to design a suitable reward function that captures the nuances of the task. However, reward engineering can be a difficult and time-consuming process. Instead, human-in-the-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human-in-the-loop RL methods still require numerous human interactions to learn successful reward functions. To improve the feedback efficiency of human-in-the-loop RL methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre-training, SDP, an approach that leverages reward-free, sub-optimal data to improve scalar- and preference-based RL algorithms. In SDP, we start by pseudo-labeling all low-quality data with the minimum environment reward. Through this process, we obtain reward labels to pre-train our reward model without requiring human labeling or preferences. This pre-training phase provides the reward model a head start in learning, enabling it to recognize that low-quality transitions should be assigned low rewards. Through extensive experiments with both simulated and human teachers, we find that SDP can at least meet, but often significantly improve, state of the art human-in-the-loop RL performance across a variety of simulated robotic tasks.