StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation

TL;DR

StablePT introduces a method that separates hard and soft prompts and uses contrastive learning to improve the stability and reliability of few-shot learning with large language models, reducing variability and enhancing performance.

Contribution

The paper proposes input separation and contrastive learning for soft prompts, significantly improving stability and accuracy in few-shot learning tasks.

Findings

Outperforms state-of-the-art by 6.97% in accuracy

Reduces standard deviation by 1.92, indicating increased stability

Demonstrates robustness across 8 diverse datasets

Abstract

Large language models have shown their ability to become effective few-shot learners with prompting, revolutionizing the paradigm of learning with data scarcity. However, this approach largely depends on the quality of prompt initialization, and always exhibits large variability among different runs. Such property makes prompt tuning highly unreliable and vulnerable to poorly constructed prompts, which limits its extension to more real-world applications. To tackle this issue, we propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by the prompt initialization. Furthermore, we optimize soft prompts with contrastive learning for utilizing class-aware information in the training process to maintain model performance. Experimental results demonstrate that \sysname outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average. Furthermore, extensive experiments underscore its robustness and stability across 8 datasets covering various tasks. Codes are available at https://github.com/lccc0528/Stable/tree/main.