Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation

TL;DR

Point Policy introduces a novel approach for robot policy learning from offline human demonstration videos, using key points to represent object states and human poses, enabling effective, morphology-agnostic manipulation across diverse tasks.

Contribution

It presents a new method that learns robot policies solely from offline videos by translating human hand poses into robot actions using key points, without requiring teleoperation data.

Findings

75% improvement over prior methods in real-world tasks

74% generalization gain to novel objects

Robust performance amidst background clutter

Abstract

Building robotic agents capable of operating across diverse environments and object types remains a significant challenge, often requiring extensive data collection. This is particularly restrictive in robotics, where each data point must be physically executed in the real world. Consequently, there is a critical need for alternative data sources for robotics and frameworks that enable learning from such data. In this work, we present Point Policy, a new method for learning robot policies exclusively from offline human demonstration videos and without any teleoperation data. Point Policy leverages state-of-the-art vision models and policy architectures to translate human hand poses into robot poses while capturing object states through semantically meaningful key points. This approach yields a morphology-agnostic representation that facilitates effective policy learning. Our experiments on 8 real-world tasks demonstrate an overall 75% absolute improvement over prior works when evaluated in identical settings as training. Further, Point Policy exhibits a 74% gain across tasks for novel object instances and is robust to significant background clutter. Videos of the robot are best viewed at https://point-policy.github.io/.