Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

TL;DR

This paper explores advanced prompt programming techniques for large language models, demonstrating that zero-shot prompts can outperform few-shot prompts and proposing methods to leverage natural language and meta-prompts for better task control.

Contribution

It introduces a new perspective on prompt functions, emphasizing natural language and meta-prompts, and provides methods to improve model interaction and evaluation beyond traditional few-shot paradigms.

Findings

0-shot prompts outperform few-shot prompts in some cases

Natural language prompts can encode nuanced intentions

Meta-prompts enable models to generate their own prompts

Abstract

Prevailing methods for mapping large generative language models to supervised tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as a case study, we show that 0-shot prompts can significantly outperform few-shot prompts. We suggest that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning. This analysis motivates rethinking the role of prompts in controlling and evaluating powerful language models. In this work, we discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We explore techniques for exploiting the capacity of narratives and cultural anchors to encode nuanced intentions and techniques for encouraging deconstruction of a problem into components before producing a verdict. Informed by this more encompassing theory of prompt programming, we also introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks. Finally, we discuss how these more general methods of interacting with language models can be incorporated into existing and future benchmarks and practical applications.