LPT: Less-overfitting Prompt Tuning for Vision-Language Model

TL;DR

This paper introduces LPT, a prompt tuning framework for vision-language models that reduces overfitting and enhances generalization by filtering visual information, preserving feature structure, and constraining output class information.

Contribution

LPT employs CLIP-guided filtering, structural preservation, and hierarchical logit constraints to effectively mitigate overfitting in prompt tuning for VLMs, improving transfer performance.

Findings

LPT significantly outperforms state-of-the-art methods on various benchmarks.

The approach enhances generalization in cross-dataset and domain transfer tasks.

Structural preservation and hierarchical logit constraints effectively reduce overfitting.

Abstract

Vision-language models (VLMs) have demonstrated exceptional generalization capabilities for downstream tasks. Due to its efficiency, prompt learning has gradually become a more effective and efficient method for transferring VLMs to downstream tasks, surpassing traditional finetuning methods. However, during the transfer process, these models are prone to severe overfitting, leading to a significant decline in generalization ability. To address this issue, we propose a framework named LPT, specifically designed for vision-language models. Specifically, we use CLIP to filter out fine-grained foreground information that may lead to overfitting, thereby guiding the prompts with basic visual concepts. Additionally, to further mitigate overfitting, we have developed a Structural Preservation (SP) constraint at the feature level, which aligns the model's overall feature space structure with the frozen CLIP, endowing the feature space with overall plasticity and enabling effective reshaping of the feature space during optimization. Moreover, we employ Hierarchical Logit (HL) constraint at the output layer to constrain the overall class information in the output, complementing the role of SP at the output end. Extensive experiments across various benchmarks (from base-to-novel, cross-dataset transfer, and domain generalization) demonstrate that our approach significantly improves generalization capability and effectively alleviates overfitting compared to state-of-the-art methods.