CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation

TL;DR

This paper critically examines CLIP's multi-object representations, revealing biases towards object size and order, and analyzes their origins and impact on zero-shot classification and image generation.

Contribution

It provides a detailed analysis of CLIP's limitations in multi-object scenarios, introduces the ComCO dataset, and traces biases to training data and process.

Findings

Text encoder favors first-mentioned objects.

Image encoder prefers larger objects.

Performance drops with caption rephrasing.

Abstract

Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-oscope.github.io.