MILE: Model-based Intervention Learning

TL;DR

MILE introduces a model-based approach to leverage expert interventions and feedback signals in imitation learning, enabling effective policy learning with minimal interventions across diverse environments.

Contribution

The paper presents a novel method that utilizes both intervention data and feedback signals to learn policies efficiently with few expert interventions.

Findings

Effective in discrete and continuous environments

Successful real-world robotic manipulation application

Validated with human subject study

Abstract

Imitation learning techniques have been shown to be highly effective in real-world control scenarios, such as robotics. However, these approaches not only suffer from compounding error issues but also require human experts to provide complete trajectories. Although there exist interactive methods where an expert oversees the robot and intervenes if needed, these extensions usually only utilize the data collected during intervention periods and ignore the feedback signal hidden in non-intervention timesteps. In this work, we create a model to formulate how the interventions occur in such cases, and show that it is possible to learn a policy with just a handful of expert interventions. Our key insight is that it is possible to get crucial information about the quality of the current state and the optimality of the chosen action from expert feedback, regardless of the presence or the absence of intervention. We evaluate our method on various discrete and continuous simulation environments, a real-world robotic manipulation task, as well as a human subject study. Videos and the code can be found at https://liralab.usc.edu/mile .