HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment

TL;DR

HiMo-CLIP enhances vision-language models by incorporating semantic hierarchy and monotonicity, enabling better handling of complex, compositional, and long-form descriptions without changing the core encoder architecture.

Contribution

It introduces a hierarchical decomposition module and a monotonicity-aware contrastive loss to improve semantic understanding in CLIP-style models.

Findings

Outperforms baselines on image-text retrieval benchmarks.

Excels with long and compositional descriptions.

Produces structured, cognitively-aligned cross-modal representations.

Abstract

Contrastive vision-language models like CLIP have achieved impressive results in image-text retrieval by aligning image and text representations in a shared embedding space. However, these models often treat text as flat sequences, limiting their ability to handle complex, compositional, and long-form descriptions. In particular, they fail to capture two essential properties of language: semantic hierarchy, which reflects the multi-level compositional structure of text, and semantic monotonicity, where richer descriptions should result in stronger alignment with visual content.To address these limitations, we propose HiMo-CLIP, a representation-level framework that enhances CLIP-style models without modifying the encoder architecture. HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, enabling flexible, batch-aware alignment across different semantic granularities, and a monotonicity-aware contrastive loss (MoLo) that jointly aligns global and component-level representations, encouraging the model to internalize semantic ordering and alignment strength as a function of textual completeness.These components work in concert to produce structured, cognitively-aligned cross-modal representations. Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines, particularly under long or compositional descriptions. The code is available at https://github.com/UnicomAI/HiMo-CLIP.