Multi-Prompt with Depth Partitioned Cross-Modal Learning

TL;DR

This paper proposes a multi-prompt learning approach called PMPO that partitions visual encoder depths and combines manual and learnable prompts to improve vision-language model generalization across various tasks.

Contribution

It introduces a novel multi-modal prompting technique that divides visual encoder depths and integrates prior information to enhance model performance.

Findings

Achieves a 79.28 harmonic mean on 11 datasets, outperforming CoOp.

Improves generalization in new class, cross-dataset, and domain scenarios.

Demonstrates significant competitiveness with state-of-the-art prompting methods.

Abstract

In recent years, soft prompt learning methods have been proposed to fine-tune large-scale vision-language pre-trained models for various downstream tasks. These methods typically combine learnable textual tokens with class tokens as input for models with frozen parameters. However, they often employ a single prompt to describe class contexts, failing to capture categories' diverse attributes adequately. This study introduces the Partitioned Multi-modal Prompt (PMPO), a multi-modal prompting technique that extends the soft prompt from a single learnable prompt to multiple prompts. Our method divides the visual encoder depths and connects learnable prompts to the separated visual depths, enabling different prompts to capture the hierarchical contextual depths of visual representations. Furthermore, to maximize the advantages of multi-prompt learning, we incorporate prior information from manually designed templates and learnable multi-prompts, thus improving the generalization capabilities of our approach. We evaluate the effectiveness of our approach on three challenging tasks: new class generalization, cross-dataset evaluation, and domain generalization. For instance, our method achieves a $79.28$ harmonic mean, averaged over 11 diverse image recognition datasets ($+7.62$ compared to CoOp), demonstrating significant competitiveness compared to state-of-the-art prompting methods.