Interactive Learning from Policy-Dependent Human Feedback

TL;DR

This paper reveals that human feedback in interactive learning depends on the learner's current policy and introduces COACH, an algorithm that effectively learns behaviors from such policy-dependent feedback, demonstrated on a robot.

Contribution

The paper challenges the assumption of policy-independent human feedback and proposes COACH, a new algorithm that converges when learning from policy-dependent feedback.

Findings

Human feedback is influenced by the learner’s current policy.

COACH converges to a local optimum in learning tasks.

Successfully applied COACH to teach multiple behaviors on a robot.

Abstract

This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner's current policy. We present empirical results that show this assumption to be false -- whether human trainers give a positive or negative feedback for a decision is influenced by the learner's current policy. Based on this insight, we introduce {\em Convergent Actor-Critic by Humans} (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.