Prompt Adaptation as a Dynamic Complement in Generative AI Systems
Eaman Jahani, Benjamin S. Manning, Joe Zhang, Hong-Yi TuYe, Mohammed Alsobay, Christos Nicolaides, Siddharth Suri, David Holtz

TL;DR
This study investigates how user prompt adaptation influences performance gains in generative AI systems, revealing its varying importance depending on task structure and the limitations of automated prompt rewriting.
Contribution
The paper introduces a comprehensive experimental analysis of prompt adaptation, highlighting its role as a dynamic complement in generative AI and its dependence on task context.
Findings
Prompt adaptation accounts for about half of performance gains in fixed, goal-oriented tasks.
In open-ended creative tasks, model capability primarily drives performance improvements.
Automated prompt rewriting can modestly improve or undermine performance depending on alignment.
Abstract
As generative AI systems rapidly improve, a key question emerges: how do users adapt to these changes, and when does such adaptation matter for realizing performance gains? Drawing on theories of dynamic capabilities and IT complements, we study prompt adaptation--how users adjust their inputs in response to evolving model behavior--using a common experimental design applied to two preregistered tasks with 3,750 total participants who submitted nearly 37,000 prompts. We show that the importance of prompt adaptation depends critically on task structure. In a task with fixed evaluation criteria and an unambiguous goal, user prompt adaptation accounts for roughly half of the performance gains from a model upgrade. In contrast, in an open-ended creative task where the space of acceptable outputs is effectively unbounded and quality is subjective, performance improvements are driven…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComplex Systems and Decision Making
