Learning a Better Initialization for Soft Prompts via Meta-Learning

TL;DR

MetaPT enhances prompt tuning for language models by using meta-learning and data clustering to find better initial prompts, leading to improved performance across multiple tasks.

Contribution

This paper introduces MetaPT, a novel meta-learning approach that leverages data clustering to improve prompt initialization for better few-shot learning.

Findings

MetaPT outperforms existing methods on seven downstream tasks.

MetaPT provides more stable and consistent performance.

Clustering pre-training data helps discover commonalities that improve prompt initialization.

Abstract

Prompt tuning (PT) is an effective approach to adapting pre-trained language models to downstream tasks. Without a good initialization, prompt tuning doesn't perform well under few-shot settings. So pre-trained prompt tuning (PPT) is proposed to initialize prompts by leveraging pre-training data. We propose MetaPT (Meta-learned Prompt Tuning) to further improve PPT's initialization by considering latent structure within the pre-training data. Specifically, we introduce the structure by first clustering pre-training data into different auxiliary tasks with unsupervised methods. Then we use these tasks to pre-train prompts with a meta-learning algorithm. Such a process can make prompts learn a better initialization by discovering commonalities among these auxiliary tasks. We evaluate our method on seven downstream tasks. Our MetaPT achieves better and more stable performance than the state-of-the-art method.