CXMArena: Unified Dataset to benchmark performance in realistic CXM Scenarios

TL;DR

CXMArena is a large-scale synthetic benchmark dataset designed to evaluate AI performance in realistic customer experience management scenarios, addressing current limitations in data scarcity and benchmark realism.

Contribution

The paper introduces CXMArena, a scalable LLM-powered pipeline for creating realistic CXM datasets and benchmarks across five operational tasks, filling a critical gap in practical evaluation tools.

Findings

State-of-the-art models achieve only 68% accuracy in article search.

Knowledge base refinement models have a low F1 score of 0.3.

Benchmark difficulty highlights the need for advanced models and solutions.

Abstract

Large Language Models (LLMs) hold immense potential for revolutionizing Customer Experience Management (CXM), particularly in contact center operations. However, evaluating their practical utility in complex operational environments is hindered by data scarcity (due to privacy concerns) and the limitations of current benchmarks. Existing benchmarks often lack realism, failing to incorporate deep knowledge base (KB) integration, real-world noise, or critical operational tasks beyond conversational fluency. To bridge this gap, we introduce CXMArena, a novel, large-scale synthetic benchmark dataset specifically designed for evaluating AI in operational CXM contexts. Given the diversity in possible contact center features, we have developed a scalable LLM-powered pipeline that simulates the brand's CXM entities that form the foundation of our datasets-such as knowledge articles including product specifications, issue taxonomies, and contact center conversations. The entities closely represent real-world distribution because of controlled noise injection (informed by domain experts) and rigorous automated validation. Building on this, we release CXMArena, which provides dedicated benchmarks targeting five important operational tasks: Knowledge Base Refinement, Intent Prediction, Agent Quality Adherence, Article Search, and Multi-turn RAG with Integrated Tools. Our baseline experiments underscore the benchmark's difficulty: even state of the art embedding and generation models achieve only 68% accuracy on article search, while standard embedding methods yield a low F1 score of 0.3 for knowledge base refinement, highlighting significant challenges for current models necessitating complex pipelines and solutions over conventional techniques.