Probing and Bridging Geometry-Interaction Cues for Affordance Reasoning in Vision Foundation Models

TL;DR

This paper investigates how vision foundation models understand affordance by analyzing their geometric and interaction perception capabilities, and demonstrates that combining these cues enhances affordance reasoning without additional training.

Contribution

It reveals that geometric and interaction cues are key to affordance understanding in VFMs and shows that their simple fusion improves affordance estimation in a zero-shot manner.

Findings

DINO encodes part-level geometric structures

Flux contains verb-conditioned spatial attention maps

Fusion of geometric and interaction cues improves affordance estimation

Abstract

What does it mean for a visual system to truly understand affordance? We argue that this understanding hinges on two complementary capacities: geometric perception, which identifies the structural parts of objects that enable interaction, and interaction perception, which models how an agent's actions engage with those parts. To test this hypothesis, we conduct a systematic probing of Visual Foundation Models (VFMs). We find that models like DINO inherently encode part-level geometric structures, while generative models like Flux contain rich, verb-conditioned spatial attention maps that serve as implicit interaction priors. Crucially, we demonstrate that these two dimensions are not merely correlated but are composable elements of affordance. By simply fusing DINO's geometric prototypes with Flux's interaction maps in a training-free and zero-shot manner, we achieve affordance estimation competitive with weakly-supervised methods. This final fusion experiment confirms that geometric and interaction perception are the fundamental building blocks of affordance understanding in VFMs, providing a mechanistic account of how perception grounds action.