Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks

TL;DR

This paper introduces KPPO, a novel prompt optimization framework that enhances language model performance on knowledge-intensive tasks by systematically integrating factual knowledge and reasoning patterns, reducing token usage.

Contribution

KPPO shifts prompt optimization from static elicitation to knowledge integration, introducing knowledge gap filling, batch evaluation, and adaptive pruning strategies for improved performance.

Findings

KPPO outperforms elicitation-based methods by ~6% on 15 benchmarks.

Reduces inference token usage by up to 29%.

Achieves comparable or better results with lower token consumption.

Abstract

While prompt optimization has emerged as a critical technique for enhancing language model performance, existing approaches primarily focus on elicitation-based strategies that search for optimal prompts to activate models' capabilities. These methods exhibit fundamental limitations when addressing knowledge-intensive tasks, as they operate within static knowledge capacity rather than providing the factual knowledge, terminology precision, and reasoning patterns required in specialized domains. To address these limitations, we propose Knowledge-Provision-based Prompt Optimization (KPPO), a framework that reformulates prompt optimization as systematic knowledge integration rather than potential elicitation. KPPO introduces three key innovations: 1) a knowledge gap filling mechanism for knowledge gap identification and targeted remediation; 2) a batch-wise candidate evaluation approach that considers both performance improvement and distributional stability; 3) an adaptive knowledge pruning strategy that balances performance and token efficiency, reducing up to 29% of inference token usage. Evaluation on 15 knowledge-intensive benchmarks from various domains demonstrates KPPO's superiority over elicitation-based methods, with an average improvement of ~6% over baselines while achieving comparable or lower token consumption.