Prompt Tuning for CLIP on the Pretrained Manifold

TL;DR

ManiPT enhances prompt tuning for CLIP by confining representations to the pretrained manifold, improving generalization and transferability across various downstream tasks under limited supervision.

Contribution

This paper introduces ManiPT, a novel prompt tuning framework that maintains representations on the pretrained manifold using cosine constraints and structural biases, reducing overfitting.

Findings

ManiPT outperforms baseline methods in unseen-class generalization.

ManiPT improves few-shot classification accuracy.

ManiPT enhances cross-dataset transfer and domain generalization.

Abstract

Prompt tuning introduces learnable prompt vectors that adapt pretrained vision-language models to downstream tasks in a parameter-efficient manner. However, under limited supervision, prompt tuning alters pretrained representations and drives downstream features away from the pretrained manifold toward directions that are unfavorable for transfer. This drift degrades generalization. To address this limitation, we propose ManiPT, a framework that performs prompt tuning on the pretrained manifold. ManiPT introduces cosine consistency constraints in both the text and image modalities to confine the learned representations within the pretrained geometric neighborhood. Furthermore, we introduce a structural bias that enforces incremental corrections, guiding the adaptation along transferable directions to mitigate reliance on shortcut learning. From a theoretical perspective, ManiPT alleviates overfitting tendencies under limited data. Our experiments cover four downstream settings: unseen-class generalization, few-shot classification, cross-dataset transfer, and domain generalization. Across these settings, ManiPT achieves higher average performance than baseline methods. Notably, ManiPT provides an explicit perspective on how prompt tuning overfits under limited supervision.