VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos

TL;DR

This paper introduces VAGNet, a novel framework for 3D affordance grounding that leverages dynamic human-object interaction videos to improve localization accuracy, supported by a new HOI video-3D dataset.

Contribution

The paper presents VAGNet, a new method that uses video sequences for functional supervision in 3D affordance grounding, and introduces PVAD, the first HOI video-3D pairing dataset.

Findings

VAGNet outperforms static-based methods on PVAD dataset.

Dynamic interaction cues improve localization accuracy.

The PVAD dataset enables better training and evaluation of affordance models.

Abstract

3D object affordance grounding aims to identify regions on 3D objects that support human-object interaction (HOI), a capability essential to embodied visual reasoning. However, most existing approaches rely on static visual or textual cues, neglecting that affordances are inherently defined by dynamic actions. As a result, they often struggle to localize the true contact regions involved in real interactions. We take a different perspective. Humans learn how to use objects by observing and imitating actions, not just by examining shapes. Motivated by this intuition, we introduce video-guided 3D affordance grounding, which leverages dynamic interaction sequences to provide functional supervision. To achieve this, we propose VAGNet, a framework that aligns video-derived interaction cues with 3D structure to resolve ambiguities that static cues cannot address. To support this new setting, we introduce PVAD, the first HOI video-3D pairing affordance dataset, providing functional supervision unavailable in prior works. Extensive experiments on PVAD show that VAGNet achieves state-of-the-art performance, significantly outperforming static-based baselines. The code and dataset will be open publicly.