From Visual Prompt Learning to Zero-Shot Transfer: Mapping Is All You Need

TL;DR

This paper introduces SeMap, a novel mapping strategy that enhances visual prompt learning and zero-shot transfer by leveraging semantic alignment, outperforming previous prompt design methods and enabling effective task transfer without fine-tuning.

Contribution

SeMap presents a new mapping approach based on semantic alignment that significantly improves visual prompt learning and zero-shot transfer capabilities.

Findings

SeMap boosts visual prompt learning performance.

SeMap achieves competitive zero-shot transfer without fine-tuning.

SeMap enables broader applications of large vision models.

Abstract

Visual prompt learning, as a newly emerged technique, leverages the knowledge learned by a large-scale pre-trained model and adapts it to downstream tasks through the usage of prompts. While previous research has focused on designing effective prompts, in this work, we argue that compared to prompt design, a good mapping strategy matters more. In this sense, we propose SeMap, a more effective mapping using the semantic alignment between the pre-trained model's knowledge and the downstream task. Our experimental results show that SeMap can largely boost the performance of visual prompt learning. Moreover, our experiments show that SeMap is capable of achieving competitive zero-shot transfer, indicating that it can perform the downstream task without any fine-tuning on the corresponding dataset. This demonstrates the potential of our proposed method to be used in a broader range of applications where the zero-shot transfer is desired. Results suggest that our proposed SeMap could lead to significant advancements in both visual prompt learning and zero-shot transfer. We hope with SeMap, we can help the community move forward to more efficient and lightweight utilization of large vision models.