VoiceAgentEval: A Dual-Dimensional Benchmark for Expert-Level Intelligent Voice-Agent Evaluation of Xbench's Professional-Aligned Series

TL;DR

OutboundEval is a comprehensive benchmark designed to evaluate large language models in expert-level outbound calling scenarios across diverse domains, addressing dataset diversity, realistic user simulation, and evaluation accuracy.

Contribution

The paper introduces OutboundEval, a structured, domain-oriented benchmark with a large-model-driven user simulator and dynamic evaluation methods for expert-level outbound AI systems.

Findings

12 state-of-the-art LLMs evaluated, revealing trade-offs between task completion and fluency.

Benchmark covers six business domains and 30 sub-scenarios with domain-adaptive metrics.

Experiments demonstrate the effectiveness of the proposed evaluation framework.

Abstract

We propose OutboundEval, a comprehensive benchmark for evaluating large language models (LLMs) in expert-level intelligent outbound calling scenarios. Unlike existing methods that suffer from three key limitations - insufficient dataset diversity and category coverage, unrealistic user simulation, and inaccurate evaluation metrics - OutboundEval addresses these issues through a structured framework. First, we design a benchmark spanning six major business domains and 30 representative sub-scenarios, each with scenario-specific process decomposition, weighted scoring, and domain-adaptive metrics. Second, we develop a large-model-driven User Simulator that generates diverse, persona-rich virtual users with realistic behaviors, emotional variability, and communication styles, providing a controlled yet authentic testing environment. Third, we introduce a dynamic evaluation method that adapts to task variations, integrating automated and human-in-the-loop assessment to measure task execution accuracy, professional knowledge application, adaptability, and user experience quality. Experiments on 12 state-of-the-art LLMs reveal distinct trade-offs between expert-level task completion and interaction fluency, offering practical insights for building reliable, human-like outbound AI systems. OutboundEval establishes a practical, extensible, and domain-oriented standard for benchmarking LLMs in professional applications.