ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions

TL;DR

ReliabilityBench is a comprehensive benchmark that evaluates the reliability of tool-using LLM agents across multiple stress conditions, emphasizing production readiness through consistency, robustness, and fault tolerance assessments.

Contribution

It introduces a unified reliability surface, action metamorphic relations, and a fault injection framework for systematic reliability evaluation of LLM agents.

Findings

Perturbations reduce success rates significantly.

Rate limiting is the most damaging fault.

Gemini 2.0 Flash achieves reliability comparable to GPT-4o at lower cost.

Abstract

Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce \textbf{ReliabilityBench}, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using $\mathrm{pass}^k$, (ii) robustness to semantically equivalent task perturbations at intensity $\epsilon$, and (iii) fault tolerance under controlled tool/API failures at intensity $\lambda$. ReliabilityBench contributes a unified reliability surface $R(k,\epsilon,\lambda)$, \textit{action metamorphic relations} that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at $\epsilon=0$ to 88.1% at $\epsilon=0.2$. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.