Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems
Ke Chen, Yufei Zhou, Xitong Zhang, Haohan Wang

TL;DR
This paper emphasizes the importance of prompt stability in large language models, proposing a stability-aware system that improves reliability and task success through iterative enhancement and stability evaluation.
Contribution
It introduces the concept of semantic stability for prompt evaluation, develops a stability-aware prompt generation system, and establishes stability as essential for effective task performance.
Findings
Improved accuracy and consistency in task outputs
Stability-aware prompts outperform traditional prompts
Stability is a necessary condition for task success
Abstract
Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In this work, we bring attention to prompt stability-the consistency of model responses across repeated executions-as a key factor for building robust and effective prompt generation systems. To quantify this, we propose semantic stability as a criterion for assessing the response consistency of prompts, and fine-tune a LLaMA-based evaluator to measure it automatically across tasks. These components have enabled us to develop the first stability-aware…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Adversarial Robustness in Machine Learning · Explainable Artificial Intelligence (XAI)
MethodsSoftmax · Attention Is All You Need · Focus
