Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback

TL;DR

This paper addresses the challenge of teaching inverse reinforcement learners with limited feedback by formalizing the problem and proposing an algorithm that combines active state selection and policy inference, validated in a synthetic driving environment.

Contribution

It introduces a formal framework for teaching with limited feedback and develops an algorithm integrating active learning and inverse reinforcement learning techniques.

Findings

The proposed algorithm effectively teaches learners with limited feedback.

It outperforms baseline methods in a synthetic driving environment.

The method successfully infers policies from minimal trajectory data.

Abstract

We study the problem of teaching via demonstrations in sequential decision-making tasks. In particular, we focus on the situation when the teacher has no access to the learner's model and policy, and the feedback from the learner is limited to trajectories that start from states selected by the teacher. The necessity to select the starting states and infer the learner's policy creates an opportunity for using the methods of inverse reinforcement learning and active learning by the teacher. In this work, we formalize the teaching process with limited feedback and propose an algorithm that solves this teaching problem. The algorithm uses a modified version of the active value-at-risk method to select the starting states, a modified maximum causal entropy algorithm to infer the policy, and the difficulty score ratio method to choose the teaching demonstrations. We test the algorithm in a synthetic car driving environment and conclude that the proposed algorithm is an effective solution when the learner's feedback is limited.