When and How Does CLIP Enable Domain and Compositional Generalization?

TL;DR

This paper investigates how CLIP's ability to generalize to new domains and compositions depends on training data diversity, revealing that shared representations are key for effective generalization.

Contribution

It systematically studies the impact of domain diversity on CLIP's generalization, providing insights into the factors that enable domain and compositional generalization.

Findings

Domain diversity is crucial for generalization.

Compositional generalization can be weaker than domain generalization.

Shared intermediate representations are essential for successful generalization.

Abstract

The remarkable generalization performance of contrastive vision-language models like CLIP is often attributed to the diversity of their training distributions. However, key questions remain unanswered: Can CLIP generalize to an entirely unseen domain when trained on a diverse mixture of domains (domain generalization)? Can it generalize to unseen classes within partially seen domains (compositional generalization)? What factors affect such generalization? To answer these questions, we trained CLIP models on systematically constructed training distributions with controlled domain diversity and object class exposure. Our experiments show that domain diversity is essential for both domain and compositional generalization, yet compositional generalization can be surprisingly weaker than domain generalization when the training distribution contains a suboptimal subset of the test domain. Through data-centric and mechanistic analyses, we find that successful generalization requires the learning of sufficiently shared representations in intermediate layers and circuits.