Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents

TL;DR

This paper introduces IN3, a benchmark for understanding implicit user intentions in language model-driven agents, and proposes a model that improves clarification and goal refinement, enhancing interaction efficiency.

Contribution

We propose IN3, a new benchmark for implicit intention understanding, and develop Mistral-Interact, a model that enhances user-agent interaction by clarifying vague instructions.

Findings

Mistral-Interact effectively identifies vague user tasks.

The approach improves goal setting and reduces redundant tool use.

Enhanced system boosts overall agent efficiency.

Abstract

Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. Although adept at devising strategies and performing tasks, these agents struggle with seeking clarification and grasping precise user intentions. To bridge this gap, we introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. Next, we propose the incorporation of model experts as the upstream in agent designs to enhance user-agent interaction. Employing IN3, we empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals before starting downstream agent task execution. Integrating it into the XAgent framework, we comprehensively evaluate the enhanced agent system regarding user instruction understanding and execution, revealing that our approach notably excels at identifying vague user tasks, recovering and summarizing critical missing information, setting precise and necessary agent execution goals, and minimizing redundant tool usage, thus boosting overall efficiency. All the data and codes are released.