VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model

TL;DR

This paper introduces VideoAfford, a multimodal large language model framework that leverages a new large-scale video dataset to improve 3D affordance grounding by incorporating dynamic interaction and spatial reasoning.

Contribution

It presents a novel video-based 3D affordance dataset and a multimodal model that integrates dynamic interaction priors and spatial-aware learning for enhanced affordance understanding.

Findings

Outperforms existing methods in 3D affordance grounding tasks.

Demonstrates strong generalization to open-world scenarios.

Effectively reasons about affordances using dynamic and spatial cues.

Abstract

3D affordance grounding aims to highlight the actionable regions on 3D objects, which is crucial for robotic manipulation. Previous research primarily focused on learning affordance knowledge from static cues such as language and images, which struggle to provide sufficient dynamic interaction context that can reveal temporal and causal cues. To alleviate this predicament, we collect a comprehensive video-based 3D affordance dataset, \textit{VIDA}, which contains 38K human-object-interaction videos covering 16 affordance types, 38 object categories, and 22K point clouds. Based on \textit{VIDA}, we propose a strong baseline: VideoAfford, which activates multimodal large language models with additional affordance segmentation capabilities, enabling both world knowledge reasoning and fine-grained affordance grounding within a unified framework. To enhance action understanding capability, we leverage a latent action encoder to extract dynamic interaction priors from HOI videos. Moreover, we introduce a \textit{spatial-aware} loss function to enable VideoAfford to obtain comprehensive 3D spatial knowledge. Extensive experimental evaluations demonstrate that our model significantly outperforms well-established methods and exhibits strong open-world generalization with affordance reasoning abilities. All datasets and code will be publicly released to advance research in this area.