Transfering Hierarchical Structure with Dual Meta Imitation Learning

TL;DR

This paper introduces Dual Meta Imitation Learning (DMIL), a hierarchical meta-learning approach that enables robots to transfer and adapt sub-skills across multiple tasks efficiently, improving few-shot learning and long-horizon task performance.

Contribution

The paper proposes DMIL, a novel hierarchical meta-imitation learning method with iterative high-level and sub-skill meta-learning, and provides theoretical convergence proof and connections to EM algorithms.

Findings

Achieves state-of-the-art few-shot imitation on Meta-world benchmark.

Demonstrates competitive performance on long-horizon Kitchen tasks.

Provides theoretical proof of convergence and EM connection for DMIL.

Abstract

Hierarchical Imitation Learning (HIL) is an effective way for robots to learn sub-skills from long-horizon unsegmented demonstrations. However, the learned hierarchical structure lacks the mechanism to transfer across multi-tasks or to new tasks, which makes them have to learn from scratch when facing a new situation. Transferring and reorganizing modular sub-skills require fast adaptation ability of the whole hierarchical structure. In this work, we propose Dual Meta Imitation Learning (DMIL), a hierarchical meta imitation learning method where the high-level network and sub-skills are iteratively meta-learned with model-agnostic meta-learning. DMIL uses the likelihood of state-action pairs from each sub-skill as the supervision for the high-level network adaptation, and use the adapted high-level network to determine different data set for each sub-skill adaptation. We theoretically prove the convergence of the iterative training process of DMIL and establish the connection between DMIL and Expectation-Maximization algorithm. Empirically, we achieve state-of-the-art few-shot imitation learning performance on the Meta-world \cite{metaworld} benchmark and competitive results on long-horizon tasks of Kitchen environments.