Common Data Properties Limit Object-Attribute Binding in CLIP

TL;DR

This paper investigates how inherent data properties like attribute density, caption completeness, and saliency bias hinder CLIP's ability to learn object-attribute bindings, revealing that data quality is crucial for improving binding performance.

Contribution

The study identifies specific data properties that limit CLIP's binding capabilities and demonstrates that addressing these properties enables near-perfect binding, unlike previous methods.

Findings

Common data properties negatively impact CLIP's binding ability.

Scaling batch size or creating hard negatives does not improve binding.

Properly curated data with identified properties leads to near-perfect binding.

Abstract

Contrastive vision-language models like CLIP are used for a large variety of applications, such as zero-shot classification or as vision encoder for multi-modal models. Despite their popularity, their representations show major limitations. For instance, CLIP models learn bag-of-words representations and, as a consequence, fail to distinguish whether an image is of ``a yellow submarine and a blue bus'' or ``a blue submarine and a yellow bus''. Previous attempts to fix this issue added hard negatives during training or modified the architecture, but failed to resolve the problem in its entirety. We suspect that the missing insights to solve the binding problem for CLIP are hidden in arguably the most important part of learning algorithms: the data. In this work, we fill this gap by rigorously identifying the influence of data properties on CLIP's ability to learn binding using a synthetic dataset. We find that common properties of natural data such as low attribute density, incomplete captions, and the saliency bias, a tendency of human captioners to describe the object that is ``most salient'' to them, have a detrimental effect on binding performance. In contrast to common belief, we find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding. Only when the data expresses our identified data properties does CLIP learn almost perfect binding.