Error Taxonomy-Guided Prompt Optimization

TL;DR

ETGPO introduces a top-down prompt optimization method that leverages error taxonomy to target common failure modes, improving efficiency and performance in large language models across various tasks.

Contribution

The paper presents a novel error taxonomy-guided approach for prompt optimization that reduces computational costs while maintaining or improving accuracy.

Findings

ETGPO achieves comparable or better accuracy than state-of-the-art methods.

ETGPO reduces token usage and evaluation budget by about two-thirds.

Effective across mathematics, question answering, and logical reasoning benchmarks.

Abstract

Automatic Prompt Optimization (APO) is a powerful approach for extracting performance from large language models without modifying their weights. Many existing methods rely on trial-and-error, testing different prompts or in-context examples until a good configuration emerges, often consuming substantial compute. Recently, natural language feedback derived from execution logs has shown promise as a way to identify how prompts can be improved. However, most prior approaches operate in a bottom-up manner, iteratively adjusting the prompt based on feedback from individual problems, which can cause them to lose the global perspective. In this work, we propose Error Taxonomy-Guided Prompt Optimization (ETGPO), a prompt optimization algorithm that adopts a top-down approach. ETGPO focuses on the global failure landscape by collecting model errors, categorizing them into a taxonomy, and augmenting the prompt with guidance targeting the most frequent failure modes. Across multiple benchmarks spanning mathematics, question answering, and logical reasoning, ETGPO achieves accuracy that is comparable to or better than state-of-the-art methods, while requiring roughly one third of the optimization-phase token usage and evaluation budget.