P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies

TL;DR

P3-PO introduces a novel environment representation using human-prescribed points propagated through vision models, significantly enhancing the generalization of robot policies to new objects and cluttered environments.

Contribution

The paper presents P3-PO, a new framework that leverages human annotations and vision models to improve out-of-distribution generalization in robot manipulation policies.

Findings

43% improvement over prior methods in real-world tasks

58% gain on new object instances

80% gain in cluttered environments

Abstract

Developing generalizable robot policies that can robustly handle varied environmental conditions and object instances remains a fundamental challenge in robot learning. While considerable efforts have focused on collecting large robot datasets and developing policy architectures to learn from such data, naively learning from visual inputs often results in brittle policies that fail to transfer beyond the training data. This work presents Prescriptive Point Priors for Policies or P3-PO, a novel framework that constructs a unique state representation of the environment leveraging recent advances in computer vision and robot learning to achieve improved out-of-distribution generalization for robot manipulation. This representation is obtained through two steps. First, a human annotator prescribes a set of semantically meaningful points on a single demonstration frame. These points are then propagated through the dataset using off-the-shelf vision models. The derived points serve as an input to state-of-the-art policy architectures for policy learning. Our experiments across four real-world tasks demonstrate an overall 43% absolute improvement over prior methods when evaluated in identical settings as training. Further, P3-PO exhibits 58% and 80% gains across tasks for new object instances and more cluttered environments respectively. Videos illustrating the robot's performance are best viewed at point-priors.github.io.