The Tyranny of Possibilities in the Design of Task-Oriented LLM Systems: A Scoping Survey
Dhruv Dhamani, Mary Lou Maher

TL;DR
This survey explores the design space of task-oriented LLM systems, proposing conjectures and a new agent-centric perspective to guide future research in system configurations, prompting techniques, and efficiency considerations.
Contribution
It introduces seven conjectures about LLM system design, proposes an agent-centric framework for prompting, and highlights gaps in efficiency evaluation, guiding future research directions.
Findings
Identified a pattern in LLM system configurations and hypothesized outcomes.
Proposed an agent-centric perspective on prompting techniques.
Highlighted the lack of focus on computational and energy efficiency.
Abstract
This scoping survey focuses on our current understanding of the design space for task-oriented LLM systems and elaborates on definitions and relationships among the available design parameters. The paper begins by defining a minimal task-oriented LLM system and exploring the design space of such systems through a thought experiment contemplating the performance of diverse LLM system configurations (involving single LLMs, single LLM-based agents, and multiple LLM-based agent systems) on a complex software development task and hypothesizes the results. We discuss a pattern in our results and formulate them into three conjectures. While these conjectures may be partly based on faulty assumptions, they provide a starting point for future research. The paper then surveys a select few design parameters: covering and organizing research in LLM augmentation, prompting techniques, and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Multi-Agent Systems and Negotiation · Auction Theory and Applications
MethodsFocus
