Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach

TL;DR

This paper introduces a simple, effective method to enhance CLIP's ability to recognize both global and local visual details by using stochastic multi-crop augmentation, addressing its bias towards global image patterns.

Contribution

The paper proposes a plug-and-play multi-crop augmentation technique that enables CLIP to better process localized visual features, improving its performance in various settings.

Findings

Improves CLIP's recognition of local details without retraining.

Enhances zero-shot, few-shot, and test-time adaptation performance.

Addresses CLIP's bias towards global image patterns effectively.

Abstract

Vision-Language Models (VLMs) like CLIP achieve cross-modal semantic alignment through contrastive learning, exhibiting robust zero-shot generalization. Traditional prompt engineering, however, predominantly relies on coarse-grained category labels, neglecting fine-grained local semantics. Existing approaches assume that VLMs inherently recognize localized visual details and attempt to enhance classification by augmenting text prompts with attribute descriptors generated by large language models. However, our systematic experiments reveal critical limitations: CLIP's strong bias toward global image patterns hinders its ability to process localized visual descriptors. To address this fundamental constraint, we propose a simple, effective, and plug-and-play solution that enables CLIP to ``See Both the Forest and the Trees." Specifically, we employ stochastic multi-crop augmentation to activate CLIP's latent capacity for localized feature analysis. By cropping only partial regions, the approach effectively constrains the model's receptive field and recalibrates its attention mechanism, thereby mitigating its inherent bias. We evaluate the proposed method under zero-shot, few-shot, and test-time adaptation settings, and extensive experiments demonstrate that D&D achieves promising performance.