Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Models

TL;DR

Agent-GWO introduces a collaborative multi-agent framework using Grey Wolf Optimizer to automatically optimize prompts and decoding hyperparameters, enhancing reasoning accuracy and stability in large language models.

Contribution

It unifies prompt templates and hyperparameters as agent configurations and employs a leader-follower mechanism for global optimization, improving over existing single-agent methods.

Findings

Consistently improves accuracy across multiple reasoning benchmarks.

Enhances stability of reasoning performance.

Demonstrates effectiveness across diverse LLM backbones.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in complex reasoning tasks, while recent prompting strategies such as Chain-of-Thought (CoT) have further elevated their performance in handling complex logical problems. Despite these advances, high-quality reasoning remains heavily reliant on manual static prompts and is sensitive to decoding configurations and task distributions, leading to performance fluctuations and limited transferability. Existing automatic prompt optimization methods typically adopt single-agent local search, failing to simultaneously optimize prompts and decoding hyperparameters within a unified framework to achieve stable global improvements. To address this limitation, we propose Agent-GWO, a dynamic prompt optimization framework for complex reasoning. Specifically, we unify prompt templates and decoding hyperparameters as inheritable agent configurations. By leveraging the leader-follower mechanism of the Grey Wolf Optimizer (GWO), we automatically select three leader agents ($\alpha$, $\beta$, and $\delta$) to guide the collaborative updates of the remaining agents, enabling iterative convergence toward robust optimal reasoning configurations that can be seamlessly integrated for inference. Extensive experiments on multiple mathematical and hybrid reasoning benchmarks across diverse LLM backbones show that Agent-GWO consistently improves accuracy and stability over existing prompt optimization methods. The code will be released publicly.