Multi-Faceted Evaluation of Tool-Augmented Dialogue Systems

TL;DR

This paper introduces TRACE and SCOPE, new tools for systematically evaluating multi-turn, tool-augmented dialogue systems, addressing complex error detection beyond user satisfaction metrics.

Contribution

The paper presents TRACE, a benchmark of synthesized dialogues, and SCOPE, an evaluation framework that detects diverse error patterns in tool-augmented conversations.

Findings

SCOPE outperforms baseline in error detection.

SCOPE effectively identifies errors where user satisfaction is misleading.

The benchmark covers diverse error cases in multi-turn dialogues.

Abstract

Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases, and SCOPE, an evaluation framework that automatically discovers diverse error patterns and evaluation rubrics in tool-augmented dialogues. Experiments show SCOPE significantly outperforms the baseline, particularly on challenging cases where user satisfaction signals are misleading.