A Unified Evaluation-Instructed Framework for Query-Dependent Prompt Optimization

TL;DR

This paper introduces a unified, performance-oriented framework for query-dependent prompt optimization that uses a learned evaluator to guide prompt rewriting, resulting in more stable and interpretable improvements across various tasks.

Contribution

It proposes a comprehensive prompt evaluation framework and an execution-free evaluator that guides query-dependent prompt rewriting, addressing limitations of prior static and black-box methods.

Findings

Evaluator achieves high accuracy in predicting prompt performance.

Optimization outperforms static and query-dependent baselines on multiple datasets.

Framework provides stable, interpretable, and model-agnostic prompt improvements.

Abstract

Most prompt-optimization methods refine a single static template, making them ineffective in complex and dynamic user scenarios. Existing query-dependent approaches rely on unstable textual feedback or black-box reward models, providing weak and uninterpretable optimization signals. More fundamentally, prompt quality itself lacks a unified, systematic definition, resulting in fragmented and unreliable evaluation signals. Our approach first establishes a performance-oriented, systematic, and comprehensive prompt evaluation framework. Furthermore, we develop and finetune an execution-free evaluator that predicts multi-dimensional quality scores directly from text. The evaluator then instructs a metric-aware optimizer that diagnoses failure modes and rewrites prompts in an interpretable, query-dependent manner. Our evaluator achieves the strongest accuracy in predicting prompt performance, and the evaluation-instructed optimization consistently surpass both static-template and query-dependent baselines across eight datasets and on three backbone models. Overall, we propose a unified, metric-grounded perspective on prompt quality, and demonstrated that our evaluation-instructed optimization pipeline delivers stable, interpretable, and model-agnostic improvements across diverse tasks.