Submodular Evaluation Subset Selection in Automatic Prompt Optimization

TL;DR

This paper introduces SESS, a submodular evaluation subset selection method for automatic prompt optimization, which improves prompt performance by selecting better evaluation subsets through a principled, theoretically-guaranteed approach.

Contribution

It proposes a novel submodular optimization framework for selecting evaluation subsets in prompt tuning, with theoretical guarantees and demonstrated empirical improvements.

Findings

SESS outperforms random and heuristic baselines in prompt optimization.

Submodular selection leads to better task performance across multiple datasets.

The method provides a principled approach with theoretical guarantees.

Abstract

Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.