Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models

TL;DR

This paper investigates how representation disentanglement affects compositional generalization in CLIP models, using a carefully synthesized dataset to evaluate true out-of-distribution performance and identify key factors for improvement.

Contribution

It introduces a novel dataset for authentic C-OoD evaluation and demonstrates that disentangled representations are crucial for CLIP's compositional generalization.

Findings

Disentanglement correlates with better C-OoD performance

Varying C-OoD generalization observed across CLIP models

Disentanglement metrics can predict generalization capabilities

Abstract

CLIP models have recently shown to exhibit Out of Distribution (OoD) generalization capabilities. However, Compositional Out of Distribution (C-OoD) generalization, which is a crucial aspect of a model's ability to understand unseen compositions of known concepts, is relatively unexplored for the CLIP models. Our goal is to address this problem and identify the factors that contribute to the C-OoD in CLIPs. We noted that previous studies regarding compositional understanding of CLIPs frequently fail to ensure that test samples are genuinely novel relative to the CLIP training data. To this end, we carefully synthesized a large and diverse dataset in the single object setting, comprising attributes for objects that are highly unlikely to be encountered in the combined training datasets of various CLIP models. This dataset enables an authentic evaluation of C-OoD generalization. Our observations reveal varying levels of C-OoD generalization across different CLIP models. We propose that the disentanglement of CLIP representations serves as a critical indicator in this context. By utilizing our synthesized datasets and other existing datasets, we assess various disentanglement metrics of text and image representations. Our study reveals that the disentanglement of image and text representations, particularly with respect to their compositional elements, plays a crucial role in improving the generalization of CLIP models in out-of-distribution settings. This finding suggests promising opportunities for advancing out-of-distribution generalization in CLIPs.