Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

TL;DR

This paper introduces JourneyBench, a new benchmark for evaluating large language model agents in customer support, focusing on policy adherence and realistic scenarios, and demonstrates that structured agents improve compliance.

Contribution

The paper presents JourneyBench, a novel benchmark with graph-based scenario generation and a new metric for policy adherence, enabling better evaluation of LLM agents in customer support tasks.

Findings

Dynamic-Prompt Agents outperform Static-Prompt Agents in policy adherence.

Smaller models like GPT-4o-mini can outperform larger models with proper structuring.

JourneyBench effectively measures policy compliance in diverse, realistic scenarios.

Abstract

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.