Model-based Offline Imitation Learning with Non-expert Data

TL;DR

This paper introduces a scalable model-based offline imitation learning framework that effectively utilizes both suboptimal and optimal datasets, outperforming behavioral cloning especially in low-data scenarios.

Contribution

It presents a novel algorithmic framework that leverages suboptimal data in offline imitation learning, with theoretical guarantees and empirical validation.

Findings

Outperforms behavioral cloning in low-data regimes

Worst-case suboptimality is linear in the time horizon

Empirically validated on simulated continuous control domains

Abstract

Although Behavioral Cloning (BC) in theory suffers compounding errors, its scalability and simplicity still makes it an attractive imitation learning algorithm. In contrast, imitation approaches with adversarial training typically does not share the same problem, but necessitates interactions with the environment. Meanwhile, most imitation learning methods only utilises optimal datasets, which could be significantly more expensive to obtain than its suboptimal counterpart. A question that arises is, can we utilise the suboptimal dataset in a principled manner, which otherwise would have been idle? We propose a scalable model-based offline imitation learning algorithmic framework that leverages datasets collected by both suboptimal and optimal policies, and show that its worst case suboptimality becomes linear in the time horizon with respect to the expert samples. We empirically validate our theoretical results and show that the proposed method \textit{always} outperforms BC in the low data regime on simulated continuous control domains