HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models

TL;DR

HarmoCLIP introduces a framework that enhances CLIP by explicitly aligning local textual and visual semantics, improving fine-grained understanding without sacrificing global coherence, leading to state-of-the-art results.

Contribution

It proposes a novel semantic supervision strategy that harmonizes global and region representations in CLIP, addressing the local-global trade-off issue.

Findings

Achieves up to 69.78% improvement in retrieval performance.

Improves Top-1 accuracy by 3.2% on bounding-box classification.

Outperforms prior methods on global and local vision-language tasks.

Abstract

Contrastive Language-Image Pre-training (CLIP) has demonstrated remarkable generalization ability and strong performance across a wide range of vision-language tasks. However, due to the lack of region-level supervision, CLIP exhibits limited fine-grained semantic understanding. Although several methods attempt to mitigate this issue, they unintentionally disrupt the global alignment, resulting in a persistent trade-off where improving local perception simultaneously degrades global coherence. In this paper, we propose HarmoCLIP, a novel framework designed to harmonize global and region representations within CLIP. We first identify that the absence of direct alignment between local textual and visual semantics is the fundamental cause of the trade-off. To address this, HarmoCLIP introduces an explicit fine-grained semantic supervision term that directly aligns textual segments with their corresponding visual regions, effectively bridging the image region space and the textual space. To further strengthen the representation capability at the local level, our method introduces a novel Region-Language Alignment supervision strategy that promotes fine-grained semantic learning without compromising global semantic consistency. Extensive experiments demonstrate that HarmoCLIP achieves state-of-the-art (improvement up to 69.78%) performance on the global task of retrieval and yields a substantial 3.2% improvement in Top-1 accuracy on the region task of bounding-box classification, consistently outperforming prior approaches while providing a balanced, efficient, and plug-and-play solution to the global-local trade-off in CLIP. Code is available at https://github.com/Erosist/HarmoCLIP.