ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models

TL;DR

This paper introduces ClarifyMT-Bench, a comprehensive benchmark for multi-turn clarification in conversational LLMs, revealing their tendency to under-clarify and proposing ClarifyAgent to improve clarification behavior.

Contribution

It presents a new multi-turn clarification benchmark grounded in a detailed ambiguity taxonomy and diverse simulated user personas, along with a novel ClarifyAgent approach to enhance clarification in LLMs.

Findings

LLMs tend to answer prematurely in multi-turn dialogues.

Performance degrades as dialogue depth increases.

ClarifyAgent significantly improves clarification robustness.

Abstract

Large language models (LLMs) are increasingly deployed as conversational assistants in open-domain, multi-turn settings, where users often provide incomplete or ambiguous information. However, existing LLM-focused clarification benchmarks primarily assume single-turn interactions or cooperative users, limiting their ability to evaluate clarification behavior in realistic settings. We introduce \textbf{ClarifyMT-Bench}, a benchmark for multi-turn clarification grounded in a five-dimensional ambiguity taxonomy and a set of six behaviorally diverse simulated user personas. Through a hybrid LLM-human pipeline, we construct 6,120 multi-turn dialogues capturing diverse ambiguity sources and interaction patterns. Evaluating ten representative LLMs uncovers a consistent under-clarification bias: LLMs tend to answer prematurely, and performance degrades as dialogue depth increases. To mitigate this, we propose \textbf{ClarifyAgent}, an agentic approach that decomposes clarification into perception, forecasting, tracking, and planning, substantially improving robustness across ambiguity conditions. ClarifyMT-Bench establishes a reproducible foundation for studying when LLMs should ask, when they should answer, and how to navigate ambiguity in real-world human-LLM interactions.