Towards the Human Global Context: Does the Vision-Language Model Really Judge Like a Human Being?

TL;DR

This paper introduces a new metric and dataset to evaluate whether vision-language models truly understand images like humans, revealing that current models focus more on local features than global context.

Contribution

The paper proposes the Equivariance Score and Human Puzzle dataset to quantitatively assess VL models' understanding of global image context, highlighting their bias towards local features.

Findings

VL models show limited understanding of global context

Models tend to focus on specific objects or shapes

Distorting images reduces models' interpretative accuracy

Abstract

As computer vision and NLP make progress, Vision-Language(VL) is becoming an important area of research. Despite the importance, evaluation metrics of the research domain is still at a preliminary stage of development. In this paper, we propose a quantitative metric "Equivariance Score" and evaluation dataset "Human Puzzle" to assess whether a VL model is understanding an image like a human. We observed that the VL model does not interpret the overall context of an input image but instead shows biases toward a specific object or shape that forms the local context. We aim to quantitatively measure a model's performance in understanding context. To verify the current existing VL model's capability, we sliced the original input image into pieces and randomly placed them, distorting the global context of the image. Our paper discusses each VL model's level of interpretation on global context and addresses how the structural characteristics influenced the results.