Textual Gradients are a Flawed Metaphor for Automatic Prompt Optimization

TL;DR

This paper critically examines textual gradient methods for automatic prompt optimization, revealing that their underlying analogy is flawed despite some performance improvements, and offers insights for future strategy development.

Contribution

The paper provides an empirical analysis showing that textual gradients are a flawed metaphor, guiding better prompt optimization approaches.

Findings

Textual gradient methods often improve performance but do not behave as the gradient analogy suggests.

The gradient analogy does not accurately explain the behavior of these methods.

Insights from experiments can inform the development of more effective prompt optimization strategies.

Abstract

A well-engineered prompt can increase the performance of large language models; automatic prompt optimization techniques aim to increase performance without requiring human effort to tune the prompts. One leading class of prompt optimization techniques introduces the analogy of textual gradients. We investigate the behavior of these textual gradient methods through a series of experiments and case studies. While such methods often result in a performance improvement, our experiments suggest that the gradient analogy does not accurately explain their behavior. Our insights may inform the selection of prompt optimization strategies, and development of new approaches.