A Dual Approach to Imitation Learning from Observations with Offline Datasets

TL;DR

This paper introduces DILO, a novel offline imitation learning algorithm that directly learns a multi-step utility function from observation-only data, avoiding intermediate models and scaling effectively to complex, high-dimensional observations.

Contribution

DILO leverages duality to learn policies from observation data without expert actions, simplifying the process and improving scalability and performance over existing methods.

Findings

DILO outperforms existing imitation learning methods on various benchmarks.

It effectively handles high-dimensional observations.

The approach scales well with complex robot morphologies.

Abstract

Demonstrations are an effective alternative to task specification for learning agents in settings where designing a reward function is difficult. However, demonstrating expert behavior in the action space of the agent becomes unwieldy when robots have complex, unintuitive morphologies. We consider the practical setting where an agent has a dataset of prior interactions with the environment and is provided with observation-only expert demonstrations. Typical learning from observations approaches have required either learning an inverse dynamics model or a discriminator as intermediate steps of training. Errors in these intermediate one-step models compound during downstream policy learning or deployment. We overcome these limitations by directly learning a multi-step utility function that quantifies how each action impacts the agent's divergence from the expert's visitation distribution. Using the principle of duality, we derive DILO (Dual Imitation Learning from Observations), an algorithm that can leverage arbitrary suboptimal data to learn imitating policies without requiring expert actions. DILO reduces the learning from observations problem to that of simply learning an actor and a critic, bearing similar complexity to vanilla offline RL. This allows DILO to gracefully scale to high dimensional observations, and demonstrate improved performance across the board. Project page (code and videos): $\href{https://hari-sikchi.github.io/dilo/}{\text{hari-sikchi.github.io/dilo/}}$