Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models

TL;DR

This paper introduces AugPT, a novel self-contained prompt tuning method for vision-language models that uses internal data augmentation and a gating mechanism to improve performance without external knowledge.

Contribution

AugPT is a new distillation-based prompt tuning approach that leverages internal augmentation and a consensus gating mechanism, avoiding reliance on external data sources.

Findings

AugPT improves model performance and generalization.

It effectively filters noisy samples during training.

AugPT outperforms existing prompt tuning methods.

Abstract

For CLIP-based prompt tuning, introducing more data as additional knowledge for enhancing fine-tuning process is proved to be an effective approach. Existing data amplification strategies for prompt tuning typically rely on external knowledge (e.g., large language models or pre-structured knowledge bases), resulting in higher costs for data collection and processing, while generally ignoring further utilization of features in image modality. To address this, we propose Augmentation-driven Prompt Tuning (AugPT), a self-contained distillation-based prompt tuning approach using only internal augmentation on raw dataset to better exploit known features. Specifically, AugPT employs self-supervised augmentation on unlabeled images in the training set, and introduces a novel gating mechanism based on consensus test, reusing the pre-trained prompt tuning backbone model to spontaneously filter noisy samples, further enhancing the quality of augmented views. Extensive experiments validate that AugPT simultaneously enhances model performance and generalization capability without using appended external knowledge. The code of AugPT is available at: https://github.com/JREion/AugPT .