Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View

TL;DR

This paper introduces a game-theoretic evaluation framework to diagnose and understand the limitations of Vision Language Models in compositional reasoning, revealing their vulnerabilities and guiding future improvements.

Contribution

It proposes a novel game-theoretic approach to systematically assess VLMs' compositional understanding, highlighting their weaknesses and providing insights for future research.

Findings

VLMs show significant vulnerabilities in compositional reasoning.

Game-theoretic evaluation reveals specific weaknesses in relation and attribute understanding.

Insights guide future development of more robust VLMs.

Abstract

Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated at https://vlms-compositionality-gametheory.github.io/.