Imitation Learning from Observations under Transition Model Disparity

TL;DR

This paper introduces a new imitation learning method that effectively handles differences in environment dynamics by training an intermediary policy to bridge the gap between expert observations and learner actions.

Contribution

The authors propose an algorithm that trains an intermediary policy to match expert transition distributions, addressing transition model disparities in imitation learning from observations.

Findings

Outperforms baselines in MuJoCo tasks with transition mismatch

Effectively handles environment dynamics differences

Scalable approach using distribution support estimation

Abstract

Learning to perform tasks by leveraging a dataset of expert observations, also known as imitation learning from observations (ILO), is an important paradigm for learning skills without access to the expert reward function or the expert actions. We consider ILO in the setting where the expert and the learner agents operate in different environments, with the source of the discrepancy being the transition dynamics model. Recent methods for scalable ILO utilize adversarial learning to match the state-transition distributions of the expert and the learner, an approach that becomes challenging when the dynamics are dissimilar. In this work, we propose an algorithm that trains an intermediary policy in the learner environment and uses it as a surrogate expert for the learner. The intermediary policy is learned such that the state transitions generated by it are close to the state transitions in the expert dataset. To derive a practical and scalable algorithm, we employ concepts from prior work on estimating the support of a probability distribution. Experiments using MuJoCo locomotion tasks highlight that our method compares favorably to the baselines for ILO with transition dynamics mismatch.