Hierarchical Cross-modal Prompt Learning for Vision-Language Models

TL;DR

This paper introduces HiCroPL, a hierarchical cross-modal prompt learning framework for vision-language models that enhances generalization by enabling bidirectional knowledge flow between text and vision modalities, leading to state-of-the-art results.

Contribution

It proposes a novel hierarchical knowledge mapper and cross-modal prompt mechanism that improve semantic alignment and generalization in vision-language models.

Findings

Achieves state-of-the-art results on 11 benchmarks.

Enhances semantic alignment between text and vision modalities.

Improves generalization across multiple downstream tasks.

Abstract

Pre-trained Vision-Language Models (VLMs) such as CLIP have shown excellent generalization abilities. However, adapting these large-scale models to downstream tasks while preserving their generalization capabilities remains challenging. Although prompt learning methods have shown promise, they suffer from two fundamental bottlenecks that limit generalization: (a) modality isolation, and (b) hierarchical semantic decay. To address these limitations, we propose HiCroPL, a Hierarchical Cross-modal Prompt Learning framework that establishes bidirectional knowledge flow between text and vision modalities, enabling them to refine their semantics mutually. HiCroPL routes knowledge flows by leveraging the complementary strengths of text and vision. In early layers, text prompts inject relatively clear semantics into visual prompts through a hierarchical knowledge mapper, enhancing the representation of low-level visual semantics. In later layers, visual prompts encoding specific task-relevant objects flow back to refine text prompts, enabling deeper alignment. Crucially, our hierarchical knowledge mapper allows representations at multi-scales to be fused, ensuring that deeper representations retain transferable shallow semantics thereby enhancing generalization. We further introduce a lightweight layer-specific knowledge proxy to enable efficient cross-modal interactions. Extensive evaluations across four tasks demonstrate HiCroPL's superior performance, achieving state-of-the-art results on 11 benchmarks with significant improvements. Code is available at: https://github.com/zzeoZheng/HiCroPL.