InPK: Infusing Prior Knowledge into Prompt for Vision-Language Models

TL;DR

InPK enhances vision-language models by infusing class-specific prior knowledge into learnable tokens and reinforcing their interaction across multiple feature levels, leading to improved zero/few-shot recognition performance.

Contribution

The paper introduces InPK, a novel method that incorporates prior knowledge into prompt tuning for VLMs, improving generalization and discriminative ability for unseen classes.

Findings

Outperforms state-of-the-art methods on 11 datasets.

Effectively captures class-specific and universal visual concepts.

Enhances zero/few-shot recognition accuracy.

Abstract

Prompt tuning has become a popular strategy for adapting Vision-Language Models (VLMs) to zero/few-shot visual recognition tasks. Some prompting techniques introduce prior knowledge due to its richness, but when learnable tokens are randomly initialized and disconnected from prior knowledge, they tend to overfit on seen classes and struggle with domain shifts for unseen ones. To address this issue, we propose the InPK model, which infuses class-specific prior knowledge into the learnable tokens during initialization, thus enabling the model to explicitly focus on class-relevant information. Furthermore, to mitigate the weakening of class information by multi-layer encoders, we continuously reinforce the interaction between learnable tokens and prior knowledge across multiple feature levels. This progressive interaction allows the learnable tokens to better capture the fine-grained differences and universal visual concepts within prior knowledge, enabling the model to extract more discriminative and generalized text features. Even for unseen classes, the learned interaction allows the model to capture their common representations and infer their appropriate positions within the existing semantic structure. Moreover, we introduce a learnable text-to-vision projection layer to accommodate the text adjustments, ensuring better alignment of visual-text semantics. Extensive experiments on 11 recognition datasets show that InPK significantly outperforms state-of-the-art methods in multiple zero/few-shot image classification tasks.