Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

TL;DR

This paper presents the TED framework for comprehensive agent evaluation, incorporating user roles, conversation quality, and automated error diagnosis to improve understanding and performance of conversational agents.

Contribution

The TED framework introduces a unified, user-aware evaluation approach with automated error analysis, advancing beyond traditional correctness metrics.

Findings

Reveals new insights into agent performance across models and user expertise levels.

Achieves 8-10% improvements on proposed metrics after error-based refinements.

Automates error diagnosis to provide actionable feedback for agent enhancement.

Abstract

Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is challenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of a unified agent evaluation approach. Moreover, they do not systematically account for the user's role nor expertise in the interaction, providing incomplete insights into the agent's performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To address this, we introduce the TED framework (Talk, Evaluate, Diagnose). (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals-such as tool signatures, and responses-as natural language grading notes, evaluated automatically with LLM-as-a-judge. We propose new metrics that capture both turn efficiency and intermediate progress of the agent complementing the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains in agent performance with peaks of 8-10% on our proposed metrics after incorporating the identified error remedies into the agent's design.