Are Large Language Models Good Prompt Optimizers?

TL;DR

This paper critically examines the effectiveness of LLMs as prompt optimizers, revealing their limitations in error reflection and prompt generation, and proposes a new paradigm called Automatic Behavior Optimization for better control.

Contribution

The study uncovers the mechanisms and limitations of LLM-based prompt optimization and introduces the novel Automatic Behavior Optimization paradigm for improved prompt refinement.

Findings

LLMs struggle to identify true errors due to bias from prior knowledge.

Even valid reflections often fail to produce effective prompts for target models.

Target models' unpredictable behaviors hinder single-step prompt refinement.

Abstract

LLM-based Automatic Prompt Optimization, which typically utilizes LLMs as Prompt Optimizers to self-reflect and refine prompts, has shown promising performance in recent studies. Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation. In this work, we conducted a comprehensive study to uncover the actual mechanism of LLM-based Prompt Optimization. Our findings reveal that the LLM optimizers struggle to identify the true causes of errors during reflection, tending to be biased by their own prior knowledge rather than genuinely reflecting on the errors. Furthermore, even when the reflection is semantically valid, the LLM optimizers often fail to generate appropriate prompts for the target models with a single prompt refinement step, partly due to the unpredictable behaviors of the target models. Based on the observations, we introduce a new "Automatic Behavior Optimization" paradigm, which directly optimizes the target model's behavior in a more controllable manner. We hope our study can inspire new directions for automatic prompt optimization development.