Teaching Language Models To Gather Information Proactively

TL;DR

This paper introduces a new task paradigm for large language models to proactively gather missing information through strategic questioning, enhancing their collaborative problem-solving capabilities in ambiguous real-world scenarios.

Contribution

The paper proposes a scalable framework and reinforcement finetuning method enabling LLMs to identify information gaps and elicit implicit user knowledge effectively.

Findings

Qwen-2.5-7B outperforms o3-mini by 18% on automatic metrics.

Human evaluators prefer the model's questions and outlines by 42% and 28%.

Proactive clarification improves LLM collaboration quality.

Abstract

Large language models (LLMs) are increasingly expected to function as collaborative partners, engaging in back-and-forth dialogue to solve complex, ambiguous problems. However, current LLMs often falter in real-world settings, defaulting to passive responses or narrow clarifications when faced with incomplete or under-specified prompts, falling short of proactively gathering the missing information that is crucial for high-quality solutions. In this work, we introduce a new task paradigm: proactive information gathering, where LLMs must identify gaps in the provided context and strategically elicit implicit user knowledge through targeted questions. To systematically study and train this capability, we design a scalable framework that generates partially specified, real-world tasks, masking key information and simulating authentic ambiguity. Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information -- such as hidden domain expertise or fine-grained requirements -- that would otherwise remain unspoken. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these results highlight the value of proactive clarification in elevating LLMs from passive text generators to genuinely collaborative thought partners.