Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

TL;DR

This paper introduces a case-aware evaluation framework for enterprise multi-turn RAG systems, focusing on operational constraints and case-specific failure modes, enabling more accurate assessment and improvement of these systems.

Contribution

It presents a novel multi-metric evaluation framework tailored for enterprise RAG systems, addressing limitations of existing benchmarks and enabling scalable, diagnostic assessment.

Findings

Generic proxy metrics are ambiguous for enterprise RAG evaluation.

The framework exposes critical tradeoffs in system performance.

Deterministic JSON prompting enables scalable batch evaluation.

Abstract

Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production monitoring. Through a comparative study of two instruction-tuned models across short and long workflows, we show that generic proxy metrics provide ambiguous signals, while the proposed framework exposes enterprise-critical tradeoffs that are actionable for system improvement.