VisMin: Visual Minimal-Change Understanding

TL;DR

VisMin introduces a challenging benchmark for visual-language models that tests their ability to understand minimal differences in images and captions, revealing current limitations and enabling targeted improvements.

Contribution

The paper presents a new benchmark, VisMin, for fine-grained understanding, along with an automatic framework for dataset creation and a large-scale training dataset for model finetuning.

Findings

Current VLMs struggle with spatial and counting understanding.

Finetuning with the generated dataset improves model performance.

Resources and benchmarks are publicly released for future research.

Abstract

Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-language models (VLMs). Existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar captions given an image. In this paper, we introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin), which requires models to predict the correct image-caption match given two images and two captions. The image pair and caption pair contain minimal changes, i.e., only one aspect changes at a time from among the following: object, attribute, count, and spatial relation. These changes test the models' understanding of objects, attributes (such as color, material, shape), counts, and spatial relationships between objects. We built an automatic framework using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. We also generate a large-scale training dataset to finetune CLIP and Idefics2, showing significant improvements in fine-grained understanding across benchmarks and in CLIP's general image-text alignment. We release all resources, including the benchmark, training data, and finetuned model checkpoints, at https://vismin.net/.