Alternating Training-based Label Smoothing Enhances Prompt Generalization

TL;DR

This paper introduces ATLaS, an alternating training-based label smoothing technique that improves the generalization of prompt tuning in vision-language models by integrating soft labels and theoretical analysis.

Contribution

It proposes a novel ATLaS method that alternates training with hard and soft labels, enhancing prompt tuning's generalization ability with theoretical insights and efficient soft label strategies.

Findings

ATLaS consistently improves prompt tuning performance.

Combining ATLaS with CSL and ISL enhances generalization.

ATLaS is compatible with existing prompt tuning methods.

Abstract

Recent advances in pre-trained vision-language models have demonstrated remarkable zero-shot generalization capabilities. To further enhance these models' adaptability to various downstream tasks, prompt tuning has emerged as a parameter-efficient fine-tuning method. However, despite its efficiency, the generalization ability of prompt remains limited. In contrast, label smoothing (LS) has been widely recognized as an effective regularization technique that prevents models from becoming over-confident and improves their generalization. This inspires us to explore the integration of LS with prompt tuning. However, we have observed that the vanilla LS even weakens the generalization ability of prompt tuning. To address this issue, we propose the Alternating Training-based Label Smoothing (ATLaS) method, which alternately trains with standard one-hot labels and soft labels generated by LS to supervise the prompt tuning. Moreover, we introduce two types of efficient offline soft labels, including Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL), to provide inter-class or instance-class relationships for prompt tuning. The theoretical properties of the proposed ATLaS method are analyzed. Extensive experiments demonstrate that the proposed ATLaS method, combined with CSL and ISL, consistently enhances the generalization performance of prompt tuning. Moreover, the proposed ATLaS method exhibits high compatibility with prevalent prompt tuning methods, enabling seamless integration into existing methods.