Multi-class Generalized Binary Search for Active Inverse Reinforcement Learning

TL;DR

This paper introduces GBS-IRL, a novel active learning algorithm for inverse reinforcement learning that improves sample efficiency by querying informative demonstrations, with proven theoretical bounds and broad applicability.

Contribution

The paper presents GBS-IRL, the first active IRL algorithm with provable sample complexity bounds, enhancing learning efficiency through informative queries.

Findings

GBS-IRL achieves sample-efficient learning with theoretical guarantees.

Applicable to various tasks and multi-class classification problems.

Framework supports integration of different human feedback types.

Abstract

This paper addresses the problem of learning a task from demonstration. We adopt the framework of inverse reinforcement learning, where tasks are represented in the form of a reward function. Our contribution is a novel active learning algorithm that enables the learning agent to query the expert for more informative demonstrations, thus leading to more sample-efficient learning. For this novel algorithm (Generalized Binary Search for Inverse Reinforcement Learning, or GBS-IRL), we provide a theoretical bound on sample complexity and illustrate its applicability on several different tasks. To our knowledge, GBS-IRL is the first active IRL algorithm with provable sample complexity bounds. We also discuss our method in light of other existing methods in the literature and its general applicability in multi-class classification problems. Finally, motivated by recent work on learning from demonstration in robots, we also discuss how different forms of human feedback can be integrated in a transparent manner in our learning framework.