Bi-Adapt: Few-shot Bimanual Adaptation for Novel Categories of 3D Objects via Semantic Correspondence

TL;DR

Bi-Adapt enables robots to perform bimanual manipulation on new, unseen object categories efficiently by leveraging vision foundation models and semantic correspondence, reducing data needs and improving generalization.

Contribution

The paper introduces Bi-Adapt, a framework that uses semantic correspondence and vision foundation models for zero-shot bimanual manipulation of novel object categories.

Findings

High success rate on benchmark tasks across categories

Effective zero-shot generalization to out-of-category objects

Validated in both simulation and real-world environments

Abstract

Bimanual manipulation is imperative yet challenging for robots to execute complex tasks, requiring coordinated collaboration between two arms. However, existing methods for bimanual manipulation often rely on costly data collection and training, struggling to generalize to unseen objects in novel categories efficiently. In this paper, we present Bi-Adapt, a novel framework designed for efficient generalization for bimanual manipulation via semantic correspondence. Bi-Adapt achieves cross-category affordance mapping by leveraging the strong capability of vision foundation models. Fine-tuning with restricted data on novel categories, Bi-Adapt exhibits notable generalization to out-of-category objects in a zero-shot manner. Extensive experiments conducted in both simulation and real-world environments validate the effectiveness of our approach and demonstrate its high efficiency, achieving a high success rate on different benchmark tasks across novel categories with limited data. Project website: https://biadapt-project.github.io/