Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning

TL;DR

This paper introduces a multi-scale cross-modal learning approach for 3D object affordance recognition and grounding, improving the localization and understanding of object functionalities in embodied AI.

Contribution

It proposes a novel cross-modal 3D representation and a stage-wise inference strategy to jointly improve affordance grounding and classification tasks.

Findings

Enhanced accuracy in affordance grounding and classification tasks.

Ability to predict full potential affordance areas at appropriate scales.

Effective coupling of grounding and classification improves overall affordance understanding.

Abstract

A core problem of Embodied AI is to learn object manipulation from observation, as humans do. To achieve this, it is important to localize 3D object affordance areas through observation such as images (3D affordance grounding) and understand their functionalities (affordance classification). Previous attempts usually tackle these two tasks separately, leading to inconsistent predictions due to lacking proper modeling of their dependency. In addition, these methods typically only ground the incomplete affordance areas depicted in images, failing to predict the full potential affordance areas, and operate at a fixed scale, resulting in difficulty in coping with affordances significantly varying in scale with respect to the whole object. To address these issues, we propose a novel approach that learns an affordance-aware 3D representation and employs a stage-wise inference strategy leveraging the dependency between grounding and classification tasks. Specifically, we first develop a cross-modal 3D representation through efficient fusion and multi-scale geometric feature propagation, enabling inference of full potential affordance areas at a suitable regional scale. Moreover, we adopt a simple two-stage prediction mechanism, effectively coupling grounding and classification for better affordance understanding. Experiments demonstrate the effectiveness of our method, showing improved performance in both affordance grounding and classification.