ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching

TL;DR

This paper introduces ABE-CLIP, a training-free method that enhances attribute-object binding in CLIP models by refining token embeddings and aligning local tokens with image patches, leading to improved compositional image-text matching.

Contribution

It proposes a novel training-free approach with semantic refinement and local alignment strategies to improve attribute binding in CLIP without additional training.

Findings

Significantly improves attribute-object binding performance.

Outperforms training-based methods on multiple datasets.

Enhances semantic precision in image-text matching.

Abstract

Contrastive Language-Image Pretraining (CLIP) has achieved remarkable performance in various multimodal tasks. However, it still struggles with compositional image-text matching, particularly in accurately associating objects with their corresponding attributes, because its inherent global representation often overlooks fine-grained semantics for attribute binding. Existing methods often require additional training or extensive hard negative sampling, yet they frequently show limited generalization to novel compositional concepts and fail to fundamentally address the drawbacks of global representations. In this paper, we propose ABE-CLIP, a novel training-free Attribute Binding Enhancement method designed to strengthen attribute-object binding in CLIP-like models. Specifically, we employ a Semantic Refinement Mechanism to refine token embeddings for both object and attribute phrases in the text, thereby mitigating attribute confusion and improving semantic precision. We further introduce a Local Token-Patch Alignment strategy that computes similarity scores between refined textual tokens and their most relevant image patches. By aggregating localized similarity scores, ABE-CLIP computes the final image-text similarity. Experiments on multiple datasets demonstrate that ABE-CLIP significantly improves attribute-object binding performance, even surpassing methods that require extensive training.