Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlation

TL;DR

This paper presents an open-vocabulary affordance detection method for 3D point clouds that uses knowledge distillation and text-point correlation, improving semantic understanding and enabling real-time robotic applications.

Contribution

It introduces a novel approach combining knowledge distillation and text-point correlation for open-vocabulary affordance detection in 3D data, surpassing previous methods.

Findings

Achieves 7.96% higher mIOU score over baselines.

Outperforms previous methods in accuracy and generalization.

Supports real-time inference for robotic manipulation.

Abstract

Affordance detection presents intricate challenges and has a wide range of robotic applications. Previous works have faced limitations such as the complexities of 3D object shapes, the wide range of potential affordances on real-world objects, and the lack of open-vocabulary support for affordance understanding. In this paper, we introduce a new open-vocabulary affordance detection method in 3D point clouds, leveraging knowledge distillation and text-point correlation. Our approach employs pre-trained 3D models through knowledge distillation to enhance feature extraction and semantic understanding in 3D point clouds. We further introduce a new text-point correlation method to learn the semantic links between point cloud features and open-vocabulary labels. The intensive experiments show that our approach outperforms previous works and adapts to new affordance labels and unseen objects. Notably, our method achieves the improvement of 7.96% mIOU score compared to the baselines. Furthermore, it offers real-time inference which is well-suitable for robotic manipulation applications.