SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

TL;DR

SREGym is a comprehensive, live benchmark platform for AI SRE agents that simulates realistic failure scenarios in cloud-native environments to evaluate agent performance.

Contribution

It introduces SREGym, a modular, extensible framework with 90 real-world failure scenarios for evaluating AI SRE agents in complex production-like settings.

Findings

Frontier agents show up to 40% variation in failure mitigation performance.

SREGym effectively models diverse failure modes and environmental noise.

The benchmark is open-source and actively used by the research community.

Abstract

AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are simulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults at different layers, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and correlated failures. SREGym is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks. SREGym currently includes 90 realistic, challenging SRE problems. We use SREGym to evaluate frontier agents and show that their capabilities varies significantly in addressing different kinds of failures, with up to 40% differences in end-to-end results. SREGym is actively maintained as an open-source project and has been used by researchers and practitioners.