PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation

TL;DR

This paper introduces PartInstruct, a large-scale benchmark dataset for fine-grained robot manipulation tasks involving object parts, aiming to improve the training and evaluation of models on part-level instructions and generalization.

Contribution

The creation of PartInstruct, the first extensive dataset with part-level annotations and demonstrations for fine-grained manipulation, enabling better training and evaluation of robot policies.

Findings

Current models struggle with grounding part concepts.

Models have difficulty predicting actions in 3D space.

Challenges remain in long-horizon part manipulation tasks.

Abstract

Fine-grained robot manipulation, such as lifting and rotating a bottle to display the label on the cap, requires robust reasoning about object parts and their relationships with intended tasks. Despite recent advances in training general-purpose robot manipulation policies guided by language instructions, there is a notable lack of large-scale datasets for fine-grained manipulation tasks with part-level instructions and diverse 3D object instances annotated with part-level labels. In this work, we introduce PartInstruct, the first large-scale benchmark for training and evaluating fine-grained robot manipulation models using part-level instructions. PartInstruct comprises 513 object instances across 14 categories, each annotated with part-level information, and 1302 fine-grained manipulation tasks organized into 16 task classes. Our training set consists of over 10,000 expert demonstrations synthesized in a 3D simulator, where each demonstration is paired with a high-level task instruction, a chain of base part-based skill instructions, and ground-truth 3D information about the object and its parts. Additionally, we designed a comprehensive test suite to evaluate the generalizability of learned policies across new states, objects, and tasks. We evaluated several state-of-the-art robot manipulation approaches, including end-to-end vision-language policy learning and bi-level planning models for robot manipulation on our benchmark. The experimental results reveal that current models struggle to robustly ground part concepts and predict actions in 3D space, and face challenges when manipulating object parts in long-horizon tasks.