Why is prompting hard? Understanding prompts on binary sequence predictors

TL;DR

This paper investigates the challenges of finding and understanding effective prompts for neural sequence predictors, highlighting the complexity and unpredictability of optimal prompts through empirical analysis.

Contribution

It introduces an empirical framework to analyze prompts, revealing that optimal prompts can be unintuitive and difficult to identify, even with exhaustive search.

Findings

Unintuitive optimal prompts can be better understood via pretraining distribution.

Exhaustive search often fails to reliably find optimal prompts.

Popular prompting methods like demonstrations can be suboptimal.

Abstract

Frontier models can be prompted or conditioned to do many tasks, but finding good prompts is not always easy, nor is understanding some performant prompts. We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal conditioning sequences can be better understood given the pretraining distribution, which is not usually available. Even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Popular prompting methods, such as using demonstrations from the targeted task, can be surprisingly suboptimal. Using the same empirical framework, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings. Taken together, this work takes an initial step towards understanding optimal prompts, from a statistical and empirical perspective that complements research on frontier models.