CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions

TL;DR

CRMArena-Pro is a comprehensive benchmark designed to evaluate LLM agents across diverse real-world business scenarios, revealing significant gaps in current AI capabilities for enterprise applications.

Contribution

The paper introduces CRMArena-Pro, a new benchmark with expert-validated tasks, multi-turn interactions, and confidentiality assessments, addressing limitations of existing benchmarks in business AI evaluation.

Findings

Top LLM agents achieve only 58% success in single-turn tasks

Performance drops to around 35% in multi-turn interactions

Agents show near-zero inherent confidentiality awareness

Abstract

While AI agents hold transformative potential in business, effective performance benchmarking is hindered by the scarcity of public, realistic business data on widely used platforms. Existing benchmarks often lack fidelity in their environments, data, and agent-user interactions, with limited coverage of diverse business scenarios and industries. To address these gaps, we introduce CRMArena-Pro, a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings. CRMArena-Pro expands on CRMArena with nineteen expert-validated tasks across sales, service, and 'configure, price, and quote' processes, for both Business-to-Business and Business-to-Customer scenarios. It distinctively incorporates multi-turn interactions guided by diverse personas and robust confidentiality awareness assessments. Experiments reveal leading LLM agents achieve only around 58% single-turn success on CRMArena-Pro, with performance dropping significantly to approximately 35% in multi-turn settings. While Workflow Execution proves more tractable for top agents (over 83% single-turn success), other evaluated business skills present greater challenges. Furthermore, agents exhibit near-zero inherent confidentiality awareness; though targeted prompting can improve this, it often compromises task performance. These findings highlight a substantial gap between current LLM capabilities and enterprise demands, underscoring the need for advancements in multi-turn reasoning, confidentiality adherence, and versatile skill acquisition.