RIVA: Leveraging LLM Agents for Reliable Configuration Drift Detection

TL;DR

RIVA is a multi-agent system that enhances the reliability of infrastructure as code verification by effectively handling erroneous tool outputs through collaborative cross-validation, significantly improving accuracy in detecting configuration drift.

Contribution

The paper introduces RIVA, a novel multi-agent framework that improves IaC verification reliability by using collaborative cross-validation to mitigate erroneous tool responses.

Findings

RIVA increases task accuracy from 27.3% to 50% with erroneous responses.

RIVA improves accuracy from 28% to 43.8% without errors.

Cross-validation enhances autonomous infrastructure verification reliability.

Abstract

Infrastructure as code (IaC) tools automate cloud provisioning but verifying that deployed systems remain consistent with the IaC specifications remains challenging. Such configuration drift occurs because of bugs in the IaC specification, manual changes, or system updates. Large language model (LLM)-based agentic AI systems can automate the analysis of large volumes of telemetry data, making them suitable for the detection of configuration drift. However, existing agentic systems implicitly assume that the tools they invoke always return correct outputs, making them vulnerable to erroneous tool responses. Since agents cannot distinguish whether an anomalous tool output reflects a real infrastructure problem or a broken tool, such errors may cause missed drift or false alarms, reducing reliability precisely when it is most needed. We introduce RIVA (Robust Infrastructure by Verification Agents), a novel multi-agent system that performs robust IaC verification even when tools produce incorrect or misleading outputs. RIVA employs two specialized agents, a verifier agent and a tool generation agent, that collaborate through iterative cross-validation, multi-perspective verification, and tool call history tracking. Evaluation on the AIOpsLab benchmark demonstrates that RIVA, in the presence of erroneous tool responses, recovers task accuracy from 27.3% when using a baseline ReAct agent to 50.0% on average. RIVA also improves task accuracy 28% to 43.8% without erroneous tool responses. Our results show that cross-validation of diverse tool calls enables more reliable autonomous infrastructure verification in production cloud environments.