Scalable Prompt Generation for Semi-supervised Learning with Language Models

TL;DR

This paper introduces automated methods for prompt and verbalizer design in semi-supervised learning with language models, significantly reducing manual effort while maintaining high performance across NLP tasks.

Contribution

It proposes two automatic prompt generation techniques and a verbalizer method, improving scalability and efficiency in semi-supervised NLP learning.

Findings

Achieved 73.2% average accuracy, surpassing previous methods by 2.52%.

Automated prompts match or outperform manual prompts in few-shot learning.

Methods are effective across multiple NLP datasets and tasks.

Abstract

Prompt-based learning methods in semi-supervised learning (SSL) settings have been shown to be effective on multiple natural language understanding (NLU) datasets and tasks in the literature. However, manually designing multiple prompts and verbalizers requires domain knowledge and human effort, making it difficult and expensive to scale across different datasets. In this paper, we propose two methods to automatically design multiple prompts and integrate automatic verbalizer in SSL settings without sacrificing performance. The first method uses various demonstration examples with learnable continuous prompt tokens to create diverse prompt models. The second method uses a varying number of soft prompt tokens to encourage language models to learn different prompts. For the verbalizer, we use the prototypical verbalizer to replace the manual one. In summary, we obtained the best average accuracy of 73.2% (a relative improvement of 2.52% over even the previous state-of-the-art SSL method with manual prompts and verbalizers) in different few-shot learning settings.