Affordance Grounding from Demonstration Video to Target Image

TL;DR

This paper introduces Afformer, a transformer-based model with self-supervised pre-training, that improves the grounding of human hand affordances from demonstration videos to target images, crucial for robotic and AR applications.

Contribution

The paper presents a novel transformer-based model and a self-supervised pre-training method that significantly enhance affordance grounding from videos to images, addressing data scarcity and fine-grained prediction challenges.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Improves performance by 37% on the OPRA dataset.

Demonstrates effective generalization across video-image discrepancies.

Abstract

Humans excel at learning from expert demonstrations and solving their own problems. To equip intelligent robots and assistants, such as AR glasses, with this ability, it is essential to ground human hand interactions (i.e., affordances) from demonstration videos and apply them to a target image like a user's AR glass view. The video-to-image affordance grounding task is challenging due to (1) the need to predict fine-grained affordances, and (2) the limited training data, which inadequately covers video-image discrepancies and negatively impacts grounding. To tackle them, we propose Affordance Transformer (Afformer), which has a fine-grained transformer-based decoder that gradually refines affordance grounding. Moreover, we introduce Mask Affordance Hand (MaskAHand), a self-supervised pre-training technique for synthesizing video-image data and simulating context changes, enhancing affordance grounding across video-image discrepancies. Afformer with MaskAHand pre-training achieves state-of-the-art performance on multiple benchmarks, including a substantial 37% improvement on the OPRA dataset. Code is made available at https://github.com/showlab/afformer.