Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding

TL;DR

This paper introduces AdaRPG, a framework that uses foundation models for part-based perception and reasoning to improve robotic manipulation of diverse articulated objects, enabling better generalization across categories.

Contribution

The paper presents a novel framework leveraging foundation models for part grounding and reasoning, along with a new part-level affordance dataset, to enhance adaptive manipulation of articulated objects.

Findings

Strong generalization across novel object categories

Effective part-based affordance inference improves manipulation success

Framework outperforms existing methods in simulation and real-world tests

Abstract

Articulated objects pose diverse manipulation challenges for robots. Since their internal structures are not directly observable, robots must adaptively explore and refine actions to generate successful manipulation trajectories. While existing works have attempted cross-category generalization in adaptive articulated object manipulation, two major challenges persist: (1) the geometric diversity of real-world articulated objects complicates visual perception and understanding, and (2) variations in object functions and mechanisms hinder the development of a unified adaptive manipulation strategy. To address these challenges, we propose AdaRPG, a novel framework that leverages foundation models to extract object parts, which exhibit greater local geometric similarity than entire objects, thereby enhancing visual affordance generalization for functional primitive skills. To support this, we construct a part-level affordance annotation dataset to train the affordance model. Additionally, AdaRPG utilizes the common knowledge embedded in foundation models to reason about complex mechanisms and generate high-level control codes that invoke primitive skill functions based on part affordance inference. Simulation and real-world experiments demonstrate AdaRPG's strong generalization ability across novel articulated object categories.