A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search

TL;DR

This paper introduces SAILOR, a learning to search approach for imitation learning that improves recovery from mistakes and outperforms behavioral cloning on visual manipulation tasks.

Contribution

It proposes a novel learning to search framework with a world model and reward model, enhancing robustness and recovery in imitation learning.

Findings

SAILOR outperforms state-of-the-art diffusion policies on multiple benchmarks.

Scaling demonstrations for behavioral cloning does not close the performance gap.

SAILOR effectively identifies failures and resists reward hacking.

Abstract

The fundamental limitation of the behavioral cloning (BC) approach to imitation learning is that it only teaches an agent what the expert did at states the expert visited. This means that when a BC agent makes a mistake which takes them out of the support of the demonstrations, they often don't know how to recover from it. In this sense, BC is akin to giving the agent the fish -- giving them dense supervision across a narrow set of states -- rather than teaching them to fish: to be able to reason independently about achieving the expert's outcome even when faced with unseen situations at test-time. In response, we explore learning to search (L2S) from expert demonstrations, i.e. learning the components required to, at test time, plan to match expert outcomes, even after making a mistake. These include (1) a world model and (2) a reward model. We carefully ablate the set of algorithmic and design decisions required to combine these and other components for stable and sample/interaction-efficient learning of recovery behavior without additional human corrections. Across a dozen visual manipulation tasks from three benchmarks, our approach SAILOR consistently out-performs state-of-the-art Diffusion Policies trained via BC on the same data. Furthermore, scaling up the amount of demonstrations used for BC by 5-10x still leaves a performance gap. We find that SAILOR can identify nuanced failures and is robust to reward hacking. Our code is available at https://github.com/arnavkj1995/SAILOR .