SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulation

TL;DR

SKIL introduces a novel framework that uses vision foundation models to automatically extract semantic keypoints, enabling robots to learn complex tasks efficiently with fewer demonstrations and high robustness to variations.

Contribution

The paper presents SKIL, a new method that leverages semantic keypoints for data-efficient, generalizable robotic imitation learning, significantly reducing sample complexity and supporting cross-embodiment learning.

Findings

Doubles baseline performance in simple tasks.

Achieves 70% success in long-horizon tasks with 30 demonstrations.

Demonstrates robustness to object and environmental variations.

Abstract

Real-world tasks such as garment manipulation and table rearrangement demand robots to perform generalizable, highly precise, and long-horizon actions. Although imitation learning has proven to be an effective approach for teaching robots new skills, large amounts of expert demonstration data are still indispensible for these complex tasks, resulting in high sample complexity and costly data collection. To address this, we propose Semantic Keypoint Imitation Learning (SKIL), a framework which automatically obtains semantic keypoints with the help of vision foundation models, and forms the descriptor of semantic keypoints that enables efficient imitation learning of complex robotic tasks with significantly lower sample complexity. In real-world experiments, SKIL doubles the performance of baseline methods in tasks such as picking a cup or mouse, while demonstrating exceptional robustness to variations in objects, environmental changes, and distractors. For long-horizon tasks like hanging a towel on a rack where previous methods fail completely, SKIL achieves a mean success rate of 70\% with as few as 30 demonstrations. Furthermore, SKIL naturally supports cross-embodiment learning due to its semantic keypoints abstraction. Our experiments demonstrate that even human videos bring considerable improvement to the learning performance. All these results demonstrate the great success of SKIL in achieving data-efficient generalizable robotic learning. Visualizations and code are available at: https://skil-robotics.github.io/SKIL-robotics/.