Selective Contrastive Learning for Weakly Supervised Affordance Grounding

TL;DR

This paper introduces a novel selective contrastive learning approach for weakly supervised affordance grounding, leveraging cross-view object discovery and contrastive objectives to improve localization of functional parts without pixel-level annotations.

Contribution

It proposes a new method that adaptively learns affordance-relevant cues at both part and object levels using contrastive objectives and cross-view object discovery, advancing weakly supervised affordance grounding.

Findings

Effective localization of affordance-relevant regions demonstrated.

Improved performance over existing weakly supervised methods.

Cross-view object discovery enhances part-level affordance cues.

Abstract

Facilitating an entity's interaction with objects requires accurately identifying parts that afford specific actions. Weakly supervised affordance grounding (WSAG) seeks to imitate human learning from third-person demonstrations, where humans intuitively grasp functional parts without needing pixel-level annotations. To achieve this, grounding is typically learned using a shared classifier across images from different perspectives, along with distillation strategies incorporating part discovery process. However, since affordance-relevant parts are not always easily distinguishable, models primarily rely on classification, often focusing on common class-specific patterns that are unrelated to affordance. To address this limitation, we move beyond isolated part-level learning by introducing selective prototypical and pixel contrastive objectives that adaptively learn affordance-relevant cues at both the part and object levels, depending on the granularity of the available information. Initially, we find the action-associated objects in both egocentric (object-focused) and exocentric (third-person example) images by leveraging CLIP. Then, by cross-referencing the discovered objects of complementary views, we excavate the precise part-level affordance clues in each perspective. By consistently learning to distinguish affordance-relevant regions from affordance-irrelevant background context, our approach effectively shifts activation from irrelevant areas toward meaningful affordance cues. Experimental results demonstrate the effectiveness of our method. Codes are available at github.com/hynnsk/SelectiveCL.