Task-Aware 3D Affordance Segmentation via 2D Guidance and Geometric Refinement

TL;DR

This paper introduces TASA, a geometry-optimized framework that combines 2D semantic cues and 3D geometric reasoning to improve scene-level affordance segmentation for embodied agents, achieving higher accuracy and efficiency.

Contribution

TASA is the first to jointly leverage 2D semantic cues and 3D geometric reasoning in a coarse-to-fine manner for 3D affordance segmentation, addressing previous limitations.

Findings

TASA outperforms baselines in accuracy on SceneFun3D.

TASA improves efficiency in scene-level affordance segmentation.

TASA effectively integrates 2D priors with 3D geometry for coherent masks.

Abstract

Understanding 3D scene-level affordances from natural language instructions is essential for enabling embodied agents to interact meaningfully in complex environments. However, this task remains challenging due to the need for semantic reasoning and spatial grounding. Existing methods mainly focus on object-level affordances or merely lift 2D predictions to 3D, neglecting rich geometric structure information in point clouds and incurring high computational costs. To address these limitations, we introduce Task-Aware 3D Scene-level Affordance segmentation (TASA), a novel geometry-optimized framework that jointly leverages 2D semantic cues and 3D geometric reasoning in a coarse-to-fine manner. To improve the affordance detection efficiency, TASA features a task-aware 2D affordance detection module to identify manipulable points from language and visual inputs, guiding the selection of task-relevant views. To fully exploit 3D geometric information, a 3D affordance refinement module is proposed to integrate 2D semantic priors with local 3D geometry, resulting in accurate and spatially coherent 3D affordance masks. Experiments on SceneFun3D demonstrate that TASA significantly outperforms the baselines in both accuracy and efficiency in scene-level affordance segmentation.