Do Prompts Solve NLP Tasks Using Natural Language?

TL;DR

This paper empirically compares three prompt types for NLP tasks using large language models, finding schema prompts generally outperform others, especially in low-data scenarios, with performance gaps narrowing as data size increases.

Contribution

It provides a systematic comparison of human-designed, schema, and null prompts across different data regimes, highlighting the effectiveness of schema prompts.

Findings

Schema prompts are most effective overall.

Performance gaps decrease with larger training data.

Prompt effectiveness varies between few-shot and full-data settings.

Abstract

Thanks to the advanced improvement of large pre-trained language models, prompt-based fine-tuning is shown to be effective on a variety of downstream tasks. Though many prompting methods have been investigated, it remains unknown which type of prompts are the most effective among three types of prompts (i.e., human-designed prompts, schema prompts and null prompts). In this work, we empirically compare the three types of prompts under both few-shot and fully-supervised settings. Our experimental results show that schema prompts are the most effective in general. Besides, the performance gaps tend to diminish when the scale of training data grows large.