A Similarity Paradigm Through Textual Regularization Without Forgetting

TL;DR

This paper introduces SPTR, a prompt learning method that uses textual regularization and a similarity paradigm to prevent forgetting generalized knowledge, thereby enhancing model robustness and generalization across diverse tasks and datasets.

Contribution

The paper proposes a novel prompt learning framework combining textual regularization with optimal transport and a similarity paradigm to improve generalization without forgetting prior knowledge.

Findings

SPTR outperforms existing prompt learning methods on multiple tasks.

It improves generalization in few-shot, cross-dataset, and domain adaptation scenarios.

The method effectively balances knowledge retention and adaptability.

Abstract

Prompt learning has emerged as a promising method for adapting pre-trained visual-language models (VLMs) to a range of downstream tasks. While optimizing the context can be effective for improving performance on specific tasks, it can often lead to poor generalization performance on unseen classes or datasets sampled from different distributions. It may be attributed to the fact that textual prompts tend to overfit downstream data distributions, leading to the forgetting of generalized knowledge derived from hand-crafted prompts. In this paper, we propose a novel method called Similarity Paradigm with Textual Regularization (SPTR) for prompt learning without forgetting. SPTR is a two-pronged design based on hand-crafted prompts that is an inseparable framework. 1) To avoid forgetting general textual knowledge, we introduce the optimal transport as a textual regularization to finely ensure approximation with hand-crafted features and tuning textual features. 2) In order to continuously unleash the general ability of multiple hand-crafted prompts, we propose a similarity paradigm for natural alignment score and adversarial alignment score to improve model robustness for generalization. Both modules share a common objective in addressing generalization issues, aiming to maximize the generalization capability derived from multiple hand-crafted prompts. Four representative tasks (i.e., non-generalization few-shot learning, base-to-novel generalization, cross-dataset generalization, domain generalization) across 11 datasets demonstrate that SPTR outperforms existing prompt learning methods.