ProbGuard: Probabilistic Runtime Monitoring for LLM Agent Safety

TL;DR

ProbGuard is a proactive runtime monitoring framework for LLM agents that predicts safety violations in advance using probabilistic models, enhancing safety in domains like autonomous driving and household robotics.

Contribution

It introduces a novel probabilistic risk prediction approach using DTMCs, with semantic constraints and PAC guarantees, to anticipate and prevent unsafe behaviors in LLM agents.

Findings

Predicts traffic violations up to 38.66 seconds in advance.

Reduces unsafe behavior by up to 65.37%.

Maintains up to 80.4% task completion.

Abstract

Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies. We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that future executions will reach unsafe states and triggers interventions when this risk exceeds a user-defined threshold. To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and provides PAC-style guarantees on the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 38.66 seconds ahead of occurrence. In embodied agent tasks, ProbGuard reduces unsafe behavior by up to 65.37% while preserving up to 80.4% task completion. ProbGuard is implemented as an extensible open-source runtime monitor integrated with the LangChain agent framework and introduces minimal runtime overhead.