Affordance Benchmark for MLLMs

TL;DR

This paper introduces A4Bench, a benchmark to evaluate how well Multimodal Large Language Models perceive affordances, revealing significant gaps compared to human understanding, especially in dynamic and contextual scenarios.

Contribution

The paper presents A4Bench, a comprehensive benchmark for assessing affordance perception in MLLMs, including new datasets for constitutive and transformative affordances, and evaluates 17 models against human performance.

Findings

Proprietary models outperform open-source models.

All models perform significantly below human levels.

Transformative affordance perception is particularly challenging.

Abstract

Affordance theory suggests that environments inherently provide action possibilities shaping perception and behavior. While Multimodal Large Language Models (MLLMs) achieve strong performance in vision-language tasks, their ability to perceive affordance, which is crucial for intuitive and safe interactions, remains underexplored. To address this, we introduce **A4Bench**, a novel benchmark designed to evaluate the affordance perception abilities of MLLMs across two dimensions: 1) Constitutive Affordance, assessing understanding of inherent object properties through 1,282 questionanswer pairs spanning nine sub-disciplines, and 2) Transformative Affordance, probing dynamic and contextual nuances (e.g., misleading, time-dependent, cultural, or individual-specific affordance) with 718 challenging question-answer pairs. We evaluate 17 MLLMs (nine proprietary and eight open-source) and compare them to human performance. Results show that proprietary models generally outperform open-source ones, yet all models perform far below humans, especially in transformative affordance. Furthermore, even top-performing models, such as Gemini-2.0-Pro (18.05% overall exact match accuracy), significantly lag behind human performance (best: 85.34%, worst: 81.25%). These findings highlight critical gaps in environmental understanding of MLLMs and provide a foundation for advancing AI systems toward more robust, context-aware interactions.