Prompt Exploration with Prompt Regression

TL;DR

This paper introduces PEPR, a framework that predicts the effectiveness of combined prompts for large language models, aiming to improve prompt selection beyond trial-and-error by modeling prompt relations.

Contribution

PEPR is a novel method that predicts prompt combination effects using individual prompt element results, facilitating systematic prompt exploration.

Findings

PEPR accurately predicts prompt effectiveness across different LLMs.

The framework outperforms baseline prompt selection methods.

PEPR enables more efficient prompt optimization for various tasks.

Abstract

In the advent of democratized usage of large language models (LLMs), there is a growing desire to systematize LLM prompt creation and selection processes beyond iterative trial-and-error. Prior works majorly focus on searching the space of prompts without accounting for relations between prompt variations. Here we propose a framework, Prompt Exploration with Prompt Regression (PEPR), to predict the effect of prompt combinations given results for individual prompt elements as well as a simple method to select an effective prompt for a given use-case. We evaluate our approach with open-source LLMs of different sizes on several different tasks.