Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism

TL;DR

This paper introduces AIM, a robot-gated IIL algorithm that adaptively requests human interventions, reducing cognitive load and improving learning efficiency by mimicking human intervention rules with a proxy Q-function.

Contribution

The paper presents a novel adaptive intervention mechanism for IIL that learns when to request human help, significantly reducing supervision effort and improving safety-critical state identification.

Findings

AIM reduces expert monitoring efforts by 40%.

The method improves learning efficiency over baseline methods.

AIM effectively identifies safety-critical states for intervention.

Abstract

Interactive Imitation Learning (IIL) allows agents to acquire desired behaviors through human interventions, but current methods impose high cognitive demands on human supervisors. We propose the Adaptive Intervention Mechanism (AIM), a novel robot-gated IIL algorithm that learns an adaptive criterion for requesting human demonstrations. AIM utilizes a proxy Q-function to mimic the human intervention rule and adjusts intervention requests based on the alignment between agent and human actions. By assigning high Q-values when the agent deviates from the expert and decreasing these values as the agent becomes proficient, the proxy Q-function enables the agent to assess the real-time alignment with the expert and request assistance when needed. Our expert-in-the-loop experiments reveal that AIM significantly reduces expert monitoring efforts in both continuous and discrete control tasks. Compared to the uncertainty-based baseline Thrifty-DAgger, our method achieves a 40% improvement in terms of human take-over cost and learning efficiency. Furthermore, AIM effectively identifies safety-critical states for expert assistance, thereby collecting higher-quality expert demonstrations and reducing overall expert data and environment interactions needed. Code and demo video are available at https://github.com/metadriverse/AIM.