DiscoverLLM: From Executing Intents to Discovering Them

TL;DR

DiscoverLLM is a framework that trains large language models to assist users in forming and discovering their intents through adaptive exploration and refinement, improving task performance and user satisfaction.

Contribution

It introduces a novel user simulator and a hierarchical intent modeling approach enabling LLMs to better collaborate with users in ambiguous scenarios.

Findings

Achieved over 10% higher task performance across benchmarks.

Reduced conversation length by up to 40%.

Improved user satisfaction and efficiency in a human study.

Abstract

To handle ambiguous and open-ended requests, Large Language Models (LLMs) are increasingly trained to interact with users to surface intents they have not yet expressed (e.g., ask clarification questions). However, users are often ambiguous because they have not yet formed their intents: they must observe and explore outcomes to discover what they want. Simply asking "what kind of tone do you want?" fails when users themselves do not know. We introduce DiscoverLLM, a novel and generalizable framework that trains LLMs to help users form and discover their intents. Central to our approach is a novel user simulator that models cognitive state with a hierarchy of intents that progressively concretize as the model surfaces relevant options -- where the degree of concretization serves as a reward signal that models can be trained to optimize. Resulting models learn to collaborate with users by adaptively diverging (i.e., explore options) when intents are unclear, and converging (i.e., refine and implement) when intents concretize. Across proposed interactive benchmarks in creative writing, technical writing, and SVG drawing, DiscoverLLM achieves over 10% higher task performance while reducing conversation length by up to 40%. In a user study with 75 human participants, DiscoverLLM improved conversation satisfaction and efficiency compared to baselines.