DAG: Unleash the Potential of Diffusion Model for Open-Vocabulary 3D Affordance Grounding

TL;DR

This paper introduces DAG, a diffusion-based framework that leverages text-to-image diffusion models to improve 3D affordance grounding, enabling better generalization and dense affordance prediction.

Contribution

It proposes a novel method using frozen diffusion model representations to extract affordance knowledge for 3D grounding, surpassing existing methods.

Findings

Outperforms well-established methods in 3D affordance grounding

Exhibits strong open-world generalization

Enables dense affordance prediction in 3D objects

Abstract

3D object affordance grounding aims to predict the touchable regions on a 3d object, which is crucial for human-object interaction, human-robot interaction, embodied perception, and robot learning. Recent advances tackle this problem via learning from demonstration images. However, these methods fail to capture the general affordance knowledge within the image, leading to poor generalization. To address this issue, we propose to use text-to-image diffusion models to extract the general affordance knowledge because we find that such models can generate semantically valid HOI images, which demonstrate that their internal representation space is highly correlated with real-world affordance concepts. Specifically, we introduce the DAG, a diffusion-based 3d affordance grounding framework, which leverages the frozen internal representations of the text-to-image diffusion model and unlocks affordance knowledge within the diffusion model to perform 3D affordance grounding. We further introduce an affordance block and a multi-source affordance decoder to endow 3D dense affordance prediction. Extensive experimental evaluations show that our model excels over well-established methods and exhibits open-world generalization.