A Survey on AgentOps: Categorization, Challenges, and Future Directions

TL;DR

This survey systematically defines anomalies in agent systems, introduces the AgentOps framework for their operation and maintenance, and discusses challenges and future directions for advancing agent system stability and security.

Contribution

The paper provides the first comprehensive framework, AgentOps, for the operation and maintenance of agent systems, including anomaly categorization and key operational stages.

Findings

Defined anomalies as intra-agent and inter-agent issues

Proposed the AgentOps framework with four key stages

Facilitated understanding of challenges and future directions

Abstract

As the reasoning capabilities of Large Language Models (LLMs) continue to advance, LLM-based agent systems offer advantages in flexibility and interpretability over traditional systems, garnering increasing attention. However, despite the widespread research interest and industrial application of agent systems, these systems, like their traditional counterparts, frequently encounter anomalies. These anomalies lead to instability and insecurity, hindering their further development. Therefore, a comprehensive and systematic approach to the operation and maintenance of agent systems is urgently needed. Unfortunately, current research on the operations of agent systems is sparse. To address this gap, we have undertaken a survey on agent system operations with the aim of establishing a clear framework for the field, defining the challenges, and facilitating further development. Specifically, this paper begins by systematically defining anomalies within agent systems, categorizing them into intra-agent anomalies and inter-agent anomalies. Next, we introduce a novel and comprehensive operational framework for agent systems, dubbed Agent System Operations (AgentOps). We provide detailed definitions and explanations of its four key stages: monitoring, anomaly detection, root cause analysis, and resolution.