How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation

TL;DR

This paper introduces EGO-Prompt, an automated framework that optimizes prompts and reasoning processes for domain-specific tasks in large language models by integrating expert knowledge, causal graphs, and textual gradients, leading to improved performance and interpretability.

Contribution

The paper presents a novel evolutionary graph optimization framework that automatically refines prompts and causal knowledge for better domain-specific reasoning in LLMs, outperforming existing methods.

Findings

EGO-Prompt achieves 7.32%-12.61% higher F1 scores than state-of-the-art methods.

Small models with EGO-Prompt reach the performance of larger models at less than 20% of the cost.

Refined domain-specific SCGs enhance interpretability and reasoning accuracy.

Abstract

Designing optimal prompts and reasoning processes for large language models (LLMs) on domain-specific tasks is both necessary and challenging in real-world applications. Determining how to integrate domain knowledge, enhance reasoning efficiency, and even provide domain experts with refined knowledge integration hints are particularly crucial yet unresolved tasks. In this research, we propose Evolutionary Graph Optimization for Prompting (EGO-Prompt), an automated framework to designing better prompts, efficient reasoning processes and providing enhanced causal-informed process. EGO-Prompt begins with a general prompt and fault-tolerant initial Semantic Causal Graph (SCG) descriptions, constructed by human experts, which is then automatically refined and optimized to guide LLM reasoning. Recognizing that expert-defined SCGs may be partial or imperfect and that their optimal integration varies across LLMs, EGO-Prompt integrates a novel causal-guided textual gradient process in two steps: first, generating nearly deterministic reasoning guidance from the SCG for each instance, and second, adapting the LLM to effectively utilize the guidance alongside the original input. The iterative optimization algorithm further refines both the SCG and the reasoning mechanism using textual gradients with ground-truth. We tested the framework on real-world public health, transportation and human behavior tasks. EGO-Prompt achieves 7.32%-12.61% higher F1 than cutting-edge methods, and allows small models to reach the performence of larger models at under 20% of the original cost. It also outputs a refined, domain-specific SCG that improves interpretability.