CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments

TL;DR

CirrusBench is a new evaluation framework for LLM-based customer service agents, using real-world cloud support data to assess both reasoning and efficiency in complex, multi-turn interactions.

Contribution

It introduces a benchmark based on authentic cloud service tickets with novel customer-centric metrics for evaluating resolution efficiency.

Findings

State-of-the-art models excel in reasoning but struggle with complex multi-turn tasks.

Models often fail to meet efficiency standards necessary for real-world customer service.

CirrusBench highlights key areas for improving LLM-based agent performance in practical settings.

Abstract

The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI