Evaluating Model Perception of Color Illusions in Photorealistic Scenes

TL;DR

This paper investigates whether vision-language models perceive color illusions similarly to humans by creating a large dataset of realistic illusions and analyzing model responses, revealing perceptual biases akin to human vision.

Contribution

It introduces RCID, a large dataset of realistic color illusions, and demonstrates that vision-language models exhibit human-like perceptual biases.

Findings

VLMs show perceptual biases similar to humans

Created a dataset of 19,000 realistic color illusions

Trained a model to distinguish human perception from pixel differences

Abstract

We study the perception of color illusions by vision-language models. Color illusion, where a person's visual system perceives color differently from actual color, is well-studied in human vision. However, it remains underexplored whether vision-language models (VLMs), trained on large-scale human data, exhibit similar perceptual biases when confronted with such color illusions. We propose an automated framework for generating color illusion images, resulting in RCID (Realistic Color Illusion Dataset), a dataset of 19,000 realistic illusion images. Our experiments show that all studied VLMs exhibit perceptual biases similar human vision. Finally, we train a model to distinguish both human perception and actual pixel differences.