ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems

TL;DR

This paper introduces ATOD, a comprehensive benchmark and evaluation framework for assessing advanced agentic task-oriented dialogue systems with capabilities like multi-goal coordination, memory, and proactivity.

Contribution

We present ATOD, a novel benchmark with synthetic dialogues and ATOD-Eval, a detailed evaluation framework for measuring agentic behaviors in dialogue systems.

Findings

ATOD enables systematic evaluation of agentic TOD capabilities.

The proposed evaluator outperforms existing methods in accuracy and efficiency.

Experiments demonstrate the effectiveness of ATOD-Eval in comprehensive assessment.

Abstract

Recent advances in task-oriented dialogue (TOD) systems, driven by large language models (LLMs) with extensive API and tool integration, have enabled conversational agents to coordinate interleaved goals, maintain long-horizon context, and act proactively through asynchronous execution. These capabilities extend beyond traditional TOD systems, yet existing benchmarks lack systematic support for evaluating such agentic behaviors. To address this gap, we introduce ATOD, a benchmark and synthetic dialogue generation pipeline that produces richly annotated conversations requiring long-term reasoning. ATOD captures key characteristics of advanced TOD, including multi-goal coordination, dependency management, memory, adaptability, and proactivity. Building on ATOD, we propose ATOD-Eval, a holistic evaluation framework that translates these dimensions into fine-grained metrics and supports reproducible offline and online evaluation. We further present a strong agentic memory-based evaluator for benchmarking on ATOD. Experiments show that ATOD-Eval enables comprehensive assessment across task completion, agentic capability, and response quality, and that the proposed evaluator offers a better accuracy-efficiency tradeoff compared to existing memory- and LLM-based approaches under this evaluation setting.