Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models

TL;DR

This paper introduces AIME-NoB, a novel algorithm that enhances imitation learning from visual observations by overcoming embodiment and demonstration knowledge barriers through online interaction and regularization, improving efficiency and robustness.

Contribution

It proposes specific solutions to address key knowledge barriers in ILfO with pretrained models and integrates them into the AIME algorithm, significantly improving performance.

Findings

AIME-NoB outperforms baseline methods in sample efficiency.

It achieves higher convergence performance on vision-based control tasks.

The approach demonstrates robustness across different benchmarks.

Abstract

Pretraining and finetuning models has become increasingly popular in decision-making. But there are still serious impediments in Imitation Learning from Observation (ILfO) with pretrained models. This study identifies two primary obstacles: the Embodiment Knowledge Barrier (EKB) and the Demonstration Knowledge Barrier (DKB). The EKB emerges due to the pretrained models' limitations in handling novel observations, which leads to inaccurate action inference. Conversely, the DKB stems from the reliance on limited demonstration datasets, restricting the model's adaptability across diverse scenarios. We propose separate solutions to overcome each barrier and apply them to Action Inference by Maximising Evidence (AIME), a state-of-the-art algorithm. This new algorithm, AIME-NoB, integrates online interactions and a data-driven regulariser to mitigate the EKB. Additionally, it uses a surrogate reward function to broaden the policy's supported states, addressing the DKB. Our experiments on vision-based control tasks from the DeepMind Control Suite and MetaWorld benchmarks show that AIME-NoB significantly improves sample efficiency and converged performance, presenting a robust framework for overcoming the challenges in ILfO with pretrained models. Code available at https://github.com/IcarusWizard/AIME-NoB.