Theoretical Analysis of Offline Imitation With Supplementary Dataset

TL;DR

This paper provides a theoretical foundation for offline imitation learning with supplementary datasets, introducing new methods that outperform traditional behavioral cloning in certain scenarios.

Contribution

It develops a theoretical analysis of NBCU and WBCU methods, showing conditions under which supplementary data can improve imitation learning performance.

Findings

NBCU can outperform or match BC in special cases despite larger imitation gaps.

WBCU, with importance sampling, can outperform BC under mild conditions.

Empirical results demonstrate WBCU's superior performance on challenging tasks.

Abstract

Behavioral cloning (BC) can recover a good policy from abundant expert data, but may fail when expert data is insufficient. This paper considers a situation where, besides the small amount of expert data, a supplementary dataset is available, which can be collected cheaply from sub-optimal policies. Imitation learning with a supplementary dataset is an emergent practical framework, but its theoretical foundation remains under-developed. To advance understanding, we first investigate a direct extension of BC, called NBCU, that learns from the union of all available data. Our analysis shows that, although NBCU suffers an imitation gap that is larger than BC in the worst case, there exist special cases where NBCU performs better than or equally well as BC. This discovery implies that noisy data can also be helpful if utilized elaborately. Therefore, we further introduce a discriminator-based importance sampling technique to re-weight the supplementary data, proposing the WBCU method. With our newly developed landscape-based analysis, we prove that WBCU can outperform BC in mild conditions. Empirical studies show that WBCU simultaneously achieves the best performance on two challenging tasks where prior state-of-the-art methods fail.