SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model

TL;DR

SeqAfford introduces a novel 3D affordance reasoning framework that interprets complex user instructions into sequential segmentation maps, leveraging multimodal large language models and a new benchmark to enhance robotic manipulation capabilities.

Contribution

The paper presents the first instruction-based 3D affordance segmentation benchmark and a multimodal large language model, SeqAfford, for sequential reasoning and segmentation of 3D objects.

Findings

SeqAfford outperforms existing methods in 3D affordance segmentation.

The model demonstrates strong generalization to open-world scenarios.

It effectively decomposes complex instructions into sequential affordance segments.

Abstract

3D affordance segmentation aims to link human instructions to touchable regions of 3D objects for embodied manipulations. Existing efforts typically adhere to single-object, single-affordance paradigms, where each affordance type or explicit instruction strictly corresponds to a specific affordance region and are unable to handle long-horizon tasks. Such a paradigm cannot actively reason about complex user intentions that often imply sequential affordances. In this paper, we introduce the Sequential 3D Affordance Reasoning task, which extends the traditional paradigm by reasoning from cumbersome user intentions and then decomposing them into a series of segmentation maps. Toward this, we construct the first instruction-based affordance segmentation benchmark that includes reasoning over both single and sequential affordances, comprising 180K instruction-point cloud pairs. Based on the benchmark, we propose our model, SeqAfford, to unlock the 3D multi-modal large language model with additional affordance segmentation abilities, which ensures reasoning with world knowledge and fine-grained affordance grounding in a cohesive framework. We further introduce a multi-granular language-point integration module to endow 3D dense prediction. Extensive experimental evaluations show that our model excels over well-established methods and exhibits open-world generalization with sequential reasoning abilities.