AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents

TL;DR

AURA is a diagnostic framework that evaluates user satisfaction with interactive planning agents by analyzing their decision-making process, beyond just task success, to improve agent design and user experience.

Contribution

We introduce AURA, a comprehensive assessment framework that diagnoses strengths and weaknesses in agent decision-making stages to better align with user satisfaction.

Findings

Agents vary in performance across behavioral stages

User satisfaction depends on both outcomes and intermediate behaviors

Limitations identified in current user simulators for task planning

Abstract

The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task planning agents have become especially prominent in realistic scenarios involving complex internal pipelines, such as context understanding, tool management, and response generation. However, existing benchmarks predominantly evaluate agent performance based on task completion as a proxy for overall effectiveness. We hypothesize that merely improving task completion is misaligned with maximizing user satisfaction, as users interact with the entire agentic process and not only the end result. To address this gap, we propose AURA, an Agent-User inteRaction Assessment framework that conceptualizes the behavioral stages of interactive task planning agents. AURA offers a comprehensive assessment of agent through a set of atomic LLM evaluation criteria, allowing researchers and practitioners to diagnose specific strengths and weaknesses within the agent's decision-making pipeline. Our analyses show that agents excel in different behavioral stages, with user satisfaction shaped by both outcomes and intermediate behaviors. We also highlight future directions, including systems that leverage multiple agents and the limitations of user simulators in task planning.