PowerCLIP: Powerset Alignment for Contrastive Pre-Training

TL;DR

PowerCLIP introduces a powerset alignment framework for contrastive vision-language pre-training, improving fine-grained understanding and compositionality by efficiently optimizing region-to-phrase alignments.

Contribution

It proposes an efficient powerset alignment method with non-linear aggregators, enabling exhaustive region-phrase matching without exponential complexity.

Findings

Outperforms state-of-the-art in zero-shot classification

Enhances compositional understanding in vision-language tasks

Demonstrates robustness across benchmarks

Abstract

Contrastive vision-language pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics that span multiple image regions. To address this limitation, we propose PowerCLIP, a novel contrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees. Since the naive powerset construction incurs exponential computational cost due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from O(2^M) to O(M) with respect to the number of regions M, while approximating the exact loss value with arbitrary precision. Our extensive experiments demonstrate that PowerCLIP outperforms state-of-the-art methods in zero-shot classification and retrieval tasks, underscoring the compositionality and robustness of our approach. Code is available at https://github.com/Masakichi210/PowerCLIP.