One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung, Ju Hwang, Cho-Jui Hsieh

TL;DR
This paper introduces a Mixture-of-Prompts approach that divides complex tasks into sub-regions, each with specialized prompts and demos, significantly improving LLM performance over single-prompt methods.
Contribution
It proposes a novel two-phase method to construct multiple expert prompts, enhancing task coverage and performance in LLM instruction tuning.
Findings
Achieves an average win rate of 81% against prior methods
Effectively covers complex problem spaces with multiple prompts
Demonstrates significant performance improvements across benchmarks
Abstract
Large Language Models (LLMs) exhibit strong generalization capabilities to novel tasks when prompted with language instructions and in-context demos. Since this ability sensitively depends on the quality of prompts, various methods have been explored to automate the instruction design. While these methods demonstrated promising results, they also restricted the searched prompt to one instruction. Such simplification significantly limits their capacity, as a single demo-free instruction might not be able to cover the entire complex problem space of the targeted task. To alleviate this issue, we adopt the Mixture-of-Expert paradigm and divide the problem space into a set of sub-regions; Each sub-region is governed by a specialized expert, equipped with both an instruction and a set of demos. A two-phase process is developed to construct the specialized expert for each region: (1) demo…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBusiness Process Modeling and Analysis · Semantic Web and Ontologies
MethodsSparse Evolutionary Training
