Automatic Prompt Generation via Adaptive Selection of Prompting Techniques

TL;DR

This paper introduces an adaptive method for automatic prompt generation that selects suitable prompting techniques based on task descriptions, improving LLM performance without requiring expert knowledge or pre-existing templates.

Contribution

It presents a novel approach that constructs a knowledge base linking task clusters with prompting techniques, enabling dynamic and automatic prompt creation for diverse tasks.

Findings

Outperforms standard prompts on 23 tasks from BIG-Bench Extra Hard.

Demonstrates superior performance over existing automatic prompt-generation tools.

Validates effectiveness across diverse task types.

Abstract

Prompt engineering is crucial for achieving reliable and effective outputs from large language models (LLMs), but its design requires specialized knowledge of prompting techniques and a deep understanding of target tasks. To address this challenge, we propose a novel method that adaptively selects task-appropriate prompting techniques based on users' abstract task descriptions and automatically generates high-quality prompts without relying on pre-existing templates or frameworks. The proposed method constructs a knowledge base that associates task clusters, characterized by semantic similarity across diverse tasks, with their corresponding prompting techniques. When users input task descriptions, the system assigns them to the most relevant task cluster and dynamically generates prompts by integrating techniques drawn from the knowledge base. An experimental evaluation of the proposed method on 23 tasks from BIG-Bench Extra Hard (BBEH) demonstrates superior performance compared with standard prompts and existing automatic prompt-generation tools, as measured by both arithmetic and harmonic mean scores. This research establishes a foundation for streamlining and standardizing prompt creation, enabling non-experts to effectively leverage LLMs.