Automatic Prompt Optimization with Prompt Distillation

TL;DR

This paper introduces DistillPrompt, a novel autoprompting method that uses large language models to automatically optimize prompts through multi-stage integration, significantly improving performance over existing methods.

Contribution

DistillPrompt is a new autoprompting approach that employs distillation, compression, and aggregation to enhance prompt optimization without gradient-based methods.

Findings

Achieved an average 20.12% improvement over existing methods.

Effective across multiple datasets for text classification and generation.

Established as a highly effective non-gradient autoprompting technique.

Abstract

Autoprompting is the process of automatically selecting optimized prompts for language models, which is gaining popularity due to the rapid development of prompt engineering driven by extensive research in the field of large language models (LLMs). This paper presents DistillPrompt -- a novel autoprompting method based on large language models that employs a multi-stage integration of task-specific information into prompts using training data. DistillPrompt utilizes distillation, compression, and aggregation operations to explore the prompt space more thoroughly. The method was tested on different datasets for text classification and generation tasks using the t-lite-instruct-0.1 language model. The results demonstrate a significant average improvement (e.g., 20.12% across the entire dataset compared to Grips) in key metrics over existing methods in the field, establishing DistillPrompt as one of the most effective non-gradient approaches in autoprompting.