SAGE: A Service Agent Graph-guided Evaluation Benchmark

TL;DR

SAGE introduces a multi-agent benchmark utilizing Dynamic Dialogue Graphs to evaluate LLMs' ability to follow structured SOPs and handle diverse user behaviors in customer service scenarios.

Contribution

It formalizes unstructured SOPs into Dynamic Dialogue Graphs and provides a modular framework for domain adaptation and automated dialogue synthesis.

Findings

Models excel at intent classification but struggle with correct subsequent actions.

High adversarial intensity reveals models' 'Empathy Resilience' despite logical failures.

Significant 'Execution Gap' observed across 27 LLMs in industrial scenarios.

Abstract

The development of Large Language Models (LLMs) has catalyzed automation in customer service, yet benchmarking their performance remains challenging. Existing benchmarks predominantly rely on static paradigms and single-dimensional metrics, failing to account for diverse user behaviors or the strict adherence to structured Standard Operating Procedures (SOPs) required in real-world deployments. To bridge this gap, we propose SAGE (Service Agent Graph-guided Evaluation), a universal multi-agent benchmark for automated, dual-axis assessment. SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance and comprehensive path coverage. We introduce an Adversarial Intent Taxonomy and a modular Extension Mechanism, enabling low-cost deployment across domains and facilitating automated dialogue data synthesis. Evaluation is conducted via a framework where Judge Agents and a Rule Engine analyze interactions between User and Service Agents to generate deterministic ground truth. Extensive experiments on 27 LLMs across 6 industrial scenarios reveal a significant ``Execution Gap'' where models accurately classify intents but fail to derive correct subsequent actions. We also observe ``Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures under high adversarial intensity. Code and resources are available at https://anonymous.4open.science/r/SAGE-Bench-4CD3/.