FMimic: Foundation Models are Fine-grained Action Learners from Human Videos

TL;DR

FMimic leverages foundation models to enable robotic systems to learn fine-grained actions directly from limited human videos, significantly improving performance in various manipulation tasks.

Contribution

This work introduces FMimic, a novel approach that uses foundation models for fine-grained action learning from minimal human video data, surpassing existing high-level plan-based methods.

Findings

Strong performance with just one human video

Outperforms other methods with five videos

Achieves over 39% improvement in multi-task RLBench experiments

Abstract

Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in foundation models, particularly Vision Language Models (VLMs), have demonstrated remarkable capabilities in visual and linguistic reasoning for VIL tasks. Despite this progress, existing approaches primarily utilize these models for learning high-level plans from human demonstrations, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck for robotic systems. In this work, we present FMimic, a novel paradigm that harnesses foundation models to directly learn generalizable skills at even fine-grained action levels, using only a limited number of human videos. Extensive experiments demonstrate that our FMimic delivers strong performance with a single human video, and significantly outperforms all other methods with five videos. Furthermore, our method exhibits significant improvements of over 39% and 29% in RLBench multi-task experiments and real-world manipulation tasks, respectively, and exceeds baselines by more than 34% in high-precision tasks and 47% in long-horizon tasks.