3D-TAFS: A Training-free Framework for 3D Affordance Segmentation

TL;DR

3D-TAFS is a novel training-free framework that combines multimodal models for 3D affordance segmentation, enabling robots to better understand and interact with objects in complex indoor environments without extensive training.

Contribution

The paper introduces 3D-TAFS, a training-free multimodal framework for 3D affordance segmentation, and presents IndoorAfford-Bench, a large-scale benchmark for evaluation.

Findings

Achieves competitive performance on IndoorAfford-Bench.

Effectively integrates 2D and 3D visual understanding with language.

Demonstrates potential for improved human-robot interaction.

Abstract

Translating high-level linguistic instructions into precise robotic actions in the physical world remains challenging, particularly when considering the feasibility of interacting with 3D objects. In this paper, we introduce 3D-TAFS, a novel training-free multimodal framework for 3D affordance segmentation. To facilitate a comprehensive evaluation of such frameworks, we present IndoorAfford-Bench, a large-scale benchmark containing 9,248 images spanning 20 diverse indoor scenes across 6 areas, supporting standardized interaction queries. In particular, our framework integrates a large multimodal model with a specialized 3D vision network, enabling a seamless fusion of 2D and 3D visual understanding with language comprehension. Extensive experiments on IndoorAfford-Bench validate the proposed 3D-TAFS's capability in handling interactive 3D affordance segmentation tasks across diverse settings, showcasing competitive performance across various metrics. Our results highlight 3D-TAFS's potential for enhancing human-robot interaction based on affordance understanding in complex indoor environments, advancing the development of more intuitive and efficient robotic frameworks for real-world applications.