Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models

TL;DR

Auto-Comp introduces an automated pipeline for creating scalable, controllable benchmarks to evaluate and analyze the compositional reasoning abilities of vision-language models, revealing universal flaws and complex interactions between context and attribute binding.

Contribution

We develop Auto-Comp, a fully automated synthetic pipeline that generates controlled benchmarks for dissecting visio-linguistic reasoning in VLMs, enabling detailed analysis of their compositional failures.

Findings

VLMs exhibit universal compositional failures in color and spatial reasoning.

Models are highly susceptible to low-entropy distractors, revealing deeper flaws.

Global scene context can both aid and hinder local attribute binding.

Abstract

Modern Vision-Language Models (VLMs) exhibit a critical flaw in compositional reasoning, often confusing "a red cube and a blue sphere" with "a blue cube and a red sphere". Disentangling the visual and linguistic roots of these failures is a fundamental challenge for robust evaluation. To enable fine-grained, controllable analysis, we introduce Auto-Comp, a fully automated and synthetic pipeline for generating scalable benchmarks. Its controllable nature is key to dissecting and isolating different reasoning skills. Auto-Comp generates paired images from Minimal (e.g., "a monitor to the left of a bicycle on a white background") and LLM-generated Contextual captions (e.g., "In a brightly lit photography studio, a monitor is positioned to the left of a bicycle"), allowing a controlled A/B test to disentangle core binding ability from visio-linguistic complexity. Our evaluation of 20 VLMs on novel benchmarks for color binding and spatial relations reveals universal compositional failures in both CLIP and SigLIP model families. Crucially, our novel "Confusion Benchmark" reveals a deeper flaw beyond simple attribute swaps: models are highly susceptible to low-entropy distractors (e.g., repeated objects or colors), demonstrating their compositional failures extend beyond known bag-of-words limitations. we uncover a surprising trade-off: visio-linguistic context, which provides global scene cues, aids spatial reasoning but simultaneously hinders local attribute binding by introducing visual clutter. We release the Auto-Comp pipeline to facilitate future benchmark creation, alongside all our generated benchmarks (https://huggingface.co/AutoComp).