InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Context

TL;DR

InfoQuest is a benchmark designed to evaluate dialogue agents' ability to handle ambiguous, open-ended requests by asking clarifying questions, revealing current limitations in models' information-seeking skills.

Contribution

The paper introduces a new multi-turn dialogue benchmark for assessing how models manage hidden context and ambiguous requests in open-ended conversations.

Findings

Proprietary models perform better but still struggle with clarification.

All models require multiple turns to infer user intent.

Models often default to generic responses without proper clarification.

Abstract

Large language models excel at following explicit instructions, but they often struggle with ambiguous or incomplete user requests, defaulting to verbose, generic responses instead of seeking clarification. We introduce InfoQuest, a multi-turn chat benchmark designed to evaluate how dialogue agents handle hidden context in open-ended user requests. This benchmark presents intentionally ambiguous scenarios that require models to engage in information-seeking dialogue by asking clarifying questions before providing appropriate responses. Our evaluation of both open and closed models reveals that, while proprietary models generally perform better, all current assistants struggle to gather critical information effectively. They often require multiple turns to infer user intent and frequently default to generic responses without proper clarification. We provide a systematic methodology for generating diverse scenarios and evaluating models' information-seeking capabilities, which can be leveraged to automatically generate data for self-improvement. We also offer insights into the current limitations of language models in handling ambiguous requests through multi-turn interactions.