Evaluating Vision-Language Models on Bistable Images

TL;DR

This paper extensively evaluates vision-language models on bistable images, revealing biases, differences from human perception, and the influence of prompts and labels, with all resources openly available.

Contribution

It provides the most comprehensive analysis to date of vision-language models' responses to bistable images, including a new dataset and insights into model biases and language influence.

Findings

Most models prefer one interpretation over another

Models show minimal variance under image manipulations

Models differ from human perception and biases

Abstract

Bistable images, also known as ambiguous or reversible images, present visual stimuli that can be seen in two distinct interpretations, though not simultaneously by the observer. In this study, we conduct the most extensive examination of vision-language models using bistable images to date. We manually gathered a dataset of 29 bistable images, along with their associated labels, and subjected them to 116 different manipulations in brightness, tint, and rotation. We evaluated twelve different models in both classification and generative tasks across six model architectures. Our findings reveal that, with the exception of models from the Idefics family and LLaVA1.5-13b, there is a pronounced preference for one interpretation over another among the models, and minimal variance under image manipulations, with few exceptions on image rotations. Additionally, we compared the model preferences with humans, noting that the models do not exhibit the same continuity biases as humans and often diverge from human initial interpretations. We also investigated the influence of variations in prompts and the use of synonymous labels, discovering that these factors significantly affect model interpretations more than image manipulations showing a higher influence of the language priors on bistable image interpretations compared to image-text training data. All code and data is open sourced.