Adaptive Teaching in Heterogeneous Agents: Balancing Surprise in Sparse Reward Scenarios

TL;DR

This paper introduces a Teacher-Student learning framework that uses the concept of surprise to adapt demonstrations for heterogeneous agents, improving learning efficiency in sparse-reward control tasks.

Contribution

It proposes a novel surprise-based method for tailoring demonstrations to heterogeneous agents, addressing capability discrepancies in Learning from Demonstration.

Findings

Enhanced learning efficiency in sparse-reward environments

Effective adaptation of demonstrations to agent capabilities

Improved control performance in heterogeneous agent scenarios

Abstract

Learning from Demonstration (LfD) can be an efficient way to train systems with analogous agents by enabling ``Student'' agents to learn from the demonstrations of the most experienced ``Teacher'' agent, instead of training their policy in parallel. However, when there are discrepancies in agent capabilities, such as divergent actuator power or joint angle constraints, naively replicating demonstrations that are out of bounds for the Student's capability can limit efficient learning. We present a Teacher-Student learning framework specifically tailored to address the challenge of heterogeneity between the Teacher and Student agents. Our framework is based on the concept of ``surprise'', inspired by its application in exploration incentivization in sparse-reward environments. Surprise is repurposed to enable the Teacher to detect and adapt to differences between itself and the Student. By focusing on maximizing its surprise in response to the environment while concurrently minimizing the Student's surprise in response to the demonstrations, the Teacher agent can effectively tailor its demonstrations to the Student's specific capabilities and constraints. We validate our method by demonstrating improvements in the Student's learning in control tasks within sparse-reward environments.