Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors

TL;DR

This paper presents a novel weakly supervised approach for affordance grounding that leverages foundation models and part-level semantic priors, significantly improving localization accuracy without dense labels.

Contribution

It introduces a new training pipeline using pseudo labels from part segmentation models, along with three key enhancements for better affordance localization.

Findings

Achieved state-of-the-art performance on affordance grounding tasks.

Demonstrated the effectiveness of part-level semantic priors in weak supervision.

Showed significant improvement over existing methods.

Abstract

In this work, we focus on the task of weakly supervised affordance grounding, where a model is trained to identify affordance regions on objects using human-object interaction images and egocentric object images without dense labels. Previous works are mostly built upon class activation maps, which are effective for semantic segmentation but may not be suitable for locating actions and functions. Leveraging recent advanced foundation models, we develop a supervised training pipeline based on pseudo labels. The pseudo labels are generated from an off-the-shelf part segmentation model, guided by a mapping from affordance to part names. Furthermore, we introduce three key enhancements to the baseline model: a label refining stage, a fine-grained feature alignment process, and a lightweight reasoning module. These techniques harness the semantic knowledge of static objects embedded in off-the-shelf foundation models to improve affordance learning, effectively bridging the gap between objects and actions. Extensive experiments demonstrate that the performance of the proposed model has achieved a breakthrough improvement over existing methods. Our codes are available at https://github.com/woyut/WSAG-PLSP .