Flow-Enabled Generalization to Human Demonstrations in Few-Shot Imitation Learning

TL;DR

This paper introduces SFCrP, a novel approach combining scene flow prediction and flow-conditioned policies to enable robots to learn from human demonstrations, improving generalization and reducing demonstration requirements.

Contribution

The paper presents SFCrP, a new method that leverages scene flow prediction and flow-conditioned policies for better generalization in imitation learning from human videos.

Findings

Outperforms state-of-the-art baselines in real-world tasks

Shows strong spatial and instance generalization to unseen scenarios

Reduces the number of robot demonstrations needed

Abstract

Imitation Learning (IL) enables robots to learn complex skills from demonstrations without explicit task modeling, but it typically requires large amounts of demonstrations, creating significant collection costs. Prior work has investigated using flow as an intermediate representation to enable the use of human videos as a substitute, thereby reducing the amount of required robot demonstrations. However, most prior work has focused on the flow, either on the object or on specific points of the robot/hand, which cannot describe the motion of interaction. Meanwhile, relying on flow to achieve generalization to scenarios observed only in human videos remains limited, as flow alone cannot capture precise motion details. Furthermore, conditioning on scene observation to produce precise actions may cause the flow-conditioned policy to overfit to training tasks and weaken the generalization indicated by the flow. To address these gaps, we propose SFCrP, which includes a Scene Flow prediction model for Cross-embodiment learning (SFCr) and a Flow and Cropped point cloud conditioned Policy (FCrP). SFCr learns from both robot and human videos and predicts any point trajectories. FCrP follows the general flow motion and adjusts the action based on observations for precision tasks. Our method outperforms SOTA baselines across various real-world task settings, while also exhibiting strong spatial and instance generalization to scenarios seen only in human videos.