Policy Contrastive Imitation Learning

TL;DR

This paper introduces Policy Contrastive Imitation Learning (PCIL), a novel approach that improves imitation learning by learning a contrastive representation space, leading to more meaningful rewards and state-of-the-art performance on challenging tasks.

Contribution

PCIL proposes a contrastive representation learning method for imitation learning, enhancing discriminator quality and reward meaningfulness, with theoretical validation and superior empirical results.

Findings

Achieves state-of-the-art performance on DeepMind Control suite

Builds a smoother, more meaningful representation space

Outperforms existing adversarial imitation learning methods

Abstract

Adversarial imitation learning (AIL) is a popular method that has recently achieved much success. However, the performance of AIL is still unsatisfactory on the more challenging tasks. We find that one of the major reasons is due to the low quality of AIL discriminator representation. Since the AIL discriminator is trained via binary classification that does not necessarily discriminate the policy from the expert in a meaningful way, the resulting reward might not be meaningful either. We propose a new method called Policy Contrastive Imitation Learning (PCIL) to resolve this issue. PCIL learns a contrastive representation space by anchoring on different policies and generates a smooth cosine-similarity-based reward. Our proposed representation learning objective can be viewed as a stronger version of the AIL objective and provide a more meaningful comparison between the agent and the policy. From a theoretical perspective, we show the validity of our method using the apprenticeship learning framework. Furthermore, our empirical evaluation on the DeepMind Control suite demonstrates that PCIL can achieve state-of-the-art performance. Finally, qualitative results suggest that PCIL builds a smoother and more meaningful representation space for imitation learning.