GPT Understands, Too

TL;DR

This paper introduces P-Tuning, a method using trainable continuous prompts combined with discrete prompts to stabilize and enhance the performance of pretrained language models across various NLU tasks.

Contribution

P-Tuning is a novel approach that employs trainable prompt embeddings to improve stability and performance in natural language understanding tasks.

Findings

P-Tuning stabilizes training performance.

It improves accuracy on NLU benchmarks like LAMA and SuperGLUE.

Effective for both frozen and fine-tuned models.

Abstract

Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.