Concurrent Training Improves the Performance of Behavioral Cloning from Observation

TL;DR

This paper introduces a modified behavioral cloning from observation (BCO*) that concurrently trains inverse dynamics and policy models, reducing initial interaction needs and achieving competitive imitation learning performance.

Contribution

The paper provides a theoretical analysis of BCO, proposes BCO* with concurrent training, and demonstrates improved sample efficiency and performance over existing methods.

Findings

BCO* reduces the dependence on initial interactions.

Concurrent training improves BCO performance.

BCO* achieves results competitive with state-of-the-art methods.

Abstract

Learning from demonstration is widely used as an efficient way for robots to acquire new skills. However, it typically requires that demonstrations provide full access to the state and action sequences. In contrast, learning from observation offers a way to utilize unlabeled demonstrations (e.g., video) to perform imitation learning. One approach to this is behavioral cloning from observation (BCO). The original implementation of BCO proceeds by first learning an inverse dynamics model and then using that model to estimate action labels, thereby reducing the problem to behavioral cloning. However, existing approaches to BCO require a large number of initial interactions in the first step. Here, we provide a novel theoretical analysis of BCO, introduce a modification BCO*, and show that in the semi-supervised setting, BCO* can concurrently improve both its estimate for the inverse dynamics model and the expert policy. This result allows us to eliminate the dependence on initial interactions and dramatically improve the sample complexity of BCO. We evaluate the effectiveness of our algorithm through experiments on various benchmark domains. The results demonstrate that concurrent training not only improves over the performance of BCO but also results in performance that is competitive with state-of-the-art imitation learning methods such as GAIL and Value-Dice.