PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations

TL;DR

This paper introduces PartManip, a large-scale benchmark and a novel learning framework for cross-category object manipulation using point cloud data, enabling better generalization to unseen objects.

Contribution

It presents the first large-scale, diverse, part-based manipulation benchmark and a cross-category policy learning method with domain adversarial training for generalization.

Findings

Our policy outperforms existing methods on unseen categories.

The approach successfully manipulates novel objects in real-world settings.

Domain adversarial learning enhances cross-category generalization.

Abstract

Learning a generalizable object manipulation policy is vital for an embodied agent to work in complex real-world scenes. Parts, as the shared components in different object categories, have the potential to increase the generalization ability of the manipulation policy and achieve cross-category object manipulation. In this work, we build the first large-scale, part-based cross-category object manipulation benchmark, PartManip, which is composed of 11 object categories, 494 objects, and 1432 tasks in 6 task classes. Compared to previous work, our benchmark is also more diverse and realistic, i.e., having more objects and using sparse-view point cloud as input without oracle information like part segmentation. To tackle the difficulties of vision-based policy learning, we first train a state-based expert with our proposed part-based canonicalization and part-aware rewards, and then distill the knowledge to a vision-based student. We also find an expressive backbone is essential to overcome the large diversity of different objects. For cross-category generalization, we introduce domain adversarial learning for domain-invariant feature extraction. Extensive experiments in simulation show that our learned policy can outperform other methods by a large margin, especially on unseen object categories. We also demonstrate our method can successfully manipulate novel objects in the real world.