CREPE: Can Vision-Language Foundation Models Reason Compositionally?

TL;DR

This paper introduces CREPE, a new benchmark to evaluate the compositional reasoning abilities of vision-language models, revealing significant struggles in systematicity and productivity across various architectures and datasets.

Contribution

The paper presents CREPE, the first comprehensive benchmark for assessing compositionality in vision-language models, highlighting their limitations in systematicity and productivity.

Findings

Models' performance drops with novel compositions, reducing Recall@1 by up to 12%.

Retrieval success declines as complexity increases, nearing random chance at high complexity.

Results are consistent across different models and training datasets.

Abstract

A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, we find that: across 7 architectures trained with 4 algorithms on massive datasets, they struggle at compositionality. To arrive at this conclusion, we introduce a new compositionality evaluation benchmark, CREPE, which measures two important aspects of compositionality identified by cognitive science literature: systematicity and productivity. To measure systematicity, CREPE consists of a test dataset containing over $370K$ image-text pairs and three different seen-unseen splits. The three splits are designed to test models trained on three popular training datasets: CC-12M, YFCC-15M, and LAION-400M. We also generate $325K$, $316K$, and $309K$ hard negative captions for a subset of the pairs. To test productivity, CREPE contains $17K$ image-text pairs with nine different complexities plus $183K$ hard negative captions with atomic, swapping and negation foils. The datasets are generated by repurposing the Visual Genome scene graphs and region descriptions and applying handcrafted templates and GPT-3. For systematicity, we find that model performance decreases consistently when novel compositions dominate the retrieval set, with Recall@1 dropping by up to $12\%$. For productivity, models' retrieval success decays as complexity increases, frequently nearing random chance at high complexity. These results hold regardless of model and training dataset size.