Hyperband-based Bayesian Optimization for Black-box Prompt Selection

TL;DR

This paper introduces HbBoPs, a method combining deep kernel Gaussian Processes with Hyperband to efficiently select optimal prompts for large language models in black-box settings, improving performance and reducing evaluation costs.

Contribution

It presents a novel prompt selection framework that integrates structural-aware Gaussian Processes with multi-fidelity Hyperband scheduling for improved efficiency.

Findings

HbBoPs outperforms existing methods in diverse benchmarks.

It reduces the number of required validation evaluations.

The approach enhances prompt selection accuracy in black-box LLM settings.

Abstract

Optimal prompt selection is crucial for maximizing large language model (LLM) performance on downstream tasks, especially in black-box settings where models are only accessible via APIs. Black-box prompt selection is challenging due to potentially large, combinatorial search spaces, absence of gradient information, and high evaluation cost of prompts on a validation set. We propose HbBoPs, a novel method that combines a structural-aware deep kernel Gaussian Process with Hyperband as a multi-fidelity scheduler to efficiently select prompts. HbBoPs uses embeddings of instructions and few-shot exemplars, treating them as modular components within prompts. This enhances the surrogate model's ability to predict which prompt to evaluate next in a sample-efficient manner. Hyperband improves query-efficiency by adaptively allocating resources across different fidelity levels, reducing the number of validation instances required for evaluating prompts. Extensive experiments across ten diverse benchmarks and three LLMs demonstrate that HbBoPs outperforms state-of-the-art methods in both performance and efficiency.