Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference

TL;DR

This paper demonstrates that improving local region-segment alignment in dual-encoder vision-language models significantly enhances their compositional generalization, especially under distribution shifts, without needing to fine-tune the entire model.

Contribution

It introduces a lightweight transformer to learn localized alignments from frozen embeddings, improving out-of-domain compositional performance over traditional fine-tuning methods.

Findings

Explicit region-segment alignment improves compositional benchmarks.

Learning localized alignment from frozen embeddings matches full fine-tuning in in-domain retrieval.

Alignment mechanisms are crucial for robust compositional generalization.

Abstract

Dual-encoder Vision-Language Models (VLMs) such as CLIP are often characterized as bag-of-words systems due to their poor performance on compositional benchmarks. We argue that this limitation may stem less from deficient representations than from the standard inference protocol based on global cosine similarity. First, through controlled diagnostic experiments, we show that explicitly enforcing fine-grained region-segment alignment at inference dramatically improves compositional performance without updating pretrained encoders. We then introduce a lightweight transformer that learns such alignments directly from frozen patch and token embeddings. Comparing against full fine-tuning and prior end-to-end compositional training methods, we find that although these approaches improve in-domain retrieval, their gains do not consistently transfer under distribution shift. In contrast, learning localized alignment over frozen representations matches full fine-tuning on in-domain retrieval while yielding substantial improvements on controlled out-of-domain compositional benchmarks. These results identify global embedding matching as a key bottleneck in dual-encoder VLMs and highlight the importance of alignment mechanisms for robust compositional generalization.