TOSC: Task-Oriented Shape Completion for Open-World Dexterous Grasp Generation from Partial Point Clouds

TL;DR

This paper introduces TOSC, a novel approach for task-oriented shape completion that improves dexterous grasping in open-world scenarios with partial data by focusing on contact regions and leveraging foundation models.

Contribution

It proposes a new task-oriented shape completion method guided by manipulation tasks, utilizing foundation models, a discriminative autoencoder, and a flow-matching grasp generator.

Findings

Achieves state-of-the-art in task-oriented grasping and shape completion.

Improves Grasp Displacement by 16.17% and Chamfer Distance by 55.26%.

Excels in grasping objects with severe missing data and generalizes across categories.

Abstract

Task-oriented dexterous grasping remains challenging in robotic manipulations of open-world objects under severe partial observation, where significant missing data invalidates generic shape completion. In this paper, to overcome this limitation, we study Task-Oriented Shape Completion, a new task that focuses on completing the potential contact regions rather than the entire shape. We argue that shape completion for grasping should be explicitly guided by the downstream manipulation task. To achieve this, we first generate multiple task-oriented shape completion candidates by leveraging the zero-shot capabilities of object functional understanding from several pre-trained foundation models. A 3D discriminative autoencoder is then proposed to evaluate the plausibility of each generated candidate and optimize the most plausible one from a global perspective. A conditional flow-matching model named FlowGrasp is developed to generate task-oriented dexterous grasps from the optimized shape. Our method achieves state-of-the-art performance in task-oriented dexterous grasping and task-oriented shape completion, improving the Grasp Displacement and the Chamfer Distance over the state-of-the-art by 16.17\% and 55.26%, respectively. In particular, it shows good capabilities in grasping objects with severe missing data. It also demonstrates good generality in handling open-set categories and tasks.