Learning from Prompt itself: the Hierarchical Attribution Prompt Optimization

TL;DR

The paper introduces HAPO, a hierarchical prompt optimization framework that improves prompt design for large language models by reducing manual effort, preventing prompt drift, and enhancing interpretability through dynamic attribution and semantic editing.

Contribution

HAPO presents a novel hierarchical optimization approach with dynamic attribution, semantic-unit editing, and multimodal support, advancing automated prompt engineering.

Findings

HAPO outperforms existing prompt optimization methods in efficiency.

HAPO maintains task performance without prompt drift.

HAPO is effective in multimodal and complex task scenarios.

Abstract

Optimization is fundamental across numerous disciplines, typically following an iterative process of refining an initial solution to enhance performance. This principle is equally critical in prompt engineering, where designing effective prompts for large language models constitutes a complex optimization challenge. A structured optimization approach requires automated or semi-automated procedures to develop improved prompts, thereby reducing manual effort, improving performance, and yielding an interpretable process. However, current prompt optimization methods often induce prompt drift, where new prompts fix prior failures but impair performance on previously successful tasks. Additionally, generating prompts from scratch can compromise interpretability. To address these limitations, this study proposes the Hierarchical Attribution Prompt Optimization (HAPO) framework, which introduces three innovations: (1) a dynamic attribution mechanism targeting error patterns in training data and prompting history, (2) semantic-unit optimization for editing functional prompt segments, and (3) multimodal-friendly progression supporting both end-to-end LLM and LLM-MLLM workflows. Applied in contexts like single/multi-image QA (e.g., OCRV2) and complex task analysis (e.g., BBH), HAPO demonstrates enhanced optimization efficiency, outperforming comparable automated prompt optimization methods and establishing an extensible paradigm for scalable prompt engineering.