Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction

TL;DR

This paper introduces a new neural network architecture for visual imitation learning that addresses the copycat problem caused by information flow issues, improving performance on high-dimensional image observations.

Contribution

The paper proposes a residual action prediction architecture that mitigates copycat problems in visual imitation learning, scalable to high-dimensional images, and validated on CARLA and MuJoCo benchmarks.

Findings

Alleviates copycat problem in visual imitation learning

Outperforms existing methods on CARLA and MuJoCo

Scales effectively to high-dimensional image data

Abstract

Imitation learning is a widely used policy learning method that enables intelligent agents to acquire complex skills from expert demonstrations. The input to the imitation learning algorithm is usually composed of both the current observation and historical observations since the most recent observation might not contain enough information. This is especially the case with image observations, where a single image only includes one view of the scene, and it suffers from a lack of motion information and object occlusions. In theory, providing multiple observations to the imitation learning agent will lead to better performance. However, surprisingly people find that sometimes imitation from observation histories performs worse than imitation from the most recent observation. In this paper, we explain this phenomenon from the information flow within the neural network perspective. We also propose a novel imitation learning neural network architecture that does not suffer from this issue by design. Furthermore, our method scales to high-dimensional image observations. Finally, we benchmark our approach on two widely used simulators, CARLA and MuJoCo, and it successfully alleviates the copycat problem and surpasses the existing solutions.