Rethinking Prompt Optimization: Reinforcement, Diversification, and Migration in Blackbox LLMs

TL;DR

This paper introduces a new prompt optimization framework for black-box LLMs that leverages positive reinforcement, feedback diversification, and prompt migration techniques to improve effectiveness, efficiency, and adaptability across models.

Contribution

It proposes a novel APO framework that incorporates positive reinforcement and feedback diversification, and formalizes continual prompt optimization for model migration.

Findings

Outperforms existing prompt optimization methods in accuracy and convergence speed.

Effectively mitigates feedback noise through diversification techniques.

Enables efficient prompt migration across different LLM versions.

Abstract

An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model outputs. While recent Automatic Prompt Optimization (APO) methods iteratively refine prompts using model-generated feedback, textual gradients, they primarily focus on error correction and neglect valuable insights from correct predictions. This limits both their effectiveness and efficiency. In this paper, we propose a novel APO framework centered on enhancing the feedback mechanism. We reinterpret the textual gradient as a form of negative reinforcement and introduce the complementary positive reinforcement to explicitly preserve beneficial prompt components identified through successful predictions. To mitigate the noise inherent in LLM-generated feedback, we introduce a technique called feedback diversification, which aggregates multiple feedback signals, emphasizing consistent, actionable advice while filtering out outliers. Motivated by the rapid evolution and diversity of available LLMs, we also formalize Continual Prompt Optimization (CPO), addressing the practical challenge of efficiently migrating optimized prompts between different model versions or API providers. Our experiments reveal that naive prompt migration often degrades performance due to loss of critical instructions. In contrast, our approach consistently outperforms strong baselines, achieving significant accuracy improvements, faster convergence, and lower computational costs in both standard and migration scenarios.