Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores

TL;DR

This paper introduces an adaptive scoring method for interactive reinforcement learning that uses human-provided scores instead of preferences, significantly improving feedback efficiency and learning near-optimal policies in robotic tasks.

Contribution

It proposes a novel adaptive learning scheme that effectively utilizes human scores for reinforcement learning, reducing feedback requirements and handling score unreliability.

Findings

Efficiently learns near-optimal policies in robotic tasks

Requires less feedback than preference-based methods

Demonstrates robustness to imperfect human scores

Abstract

Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of a large amount of interactive feedback. This paper presents a new method that uses scores provided by humans instead of pairwise preferences to improve the feedback efficiency of interactive reinforcement learning. Our key insight is that scores can yield significantly more data than pairwise preferences. Specifically, we require a teacher to interactively score the full trajectories of an agent to train a behavioral policy in a sparse reward environment. To avoid unstable scores given by humans negatively impacting the training process, we propose an adaptive learning scheme. This enables the learning paradigm to be insensitive to imperfect or unreliable scores. We extensively evaluate our method for robotic locomotion and manipulation tasks. The results show that the proposed method can efficiently learn near-optimal policies by adaptive learning from scores while requiring less feedback compared to pairwise preference learning methods. The source codes are publicly available at https://github.com/SSKKai/Interactive-Scoring-IRL.