AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents

TL;DR

AgentAuditor is a novel, training-free framework that enhances LLM-based evaluators' ability to reliably assess safety and security of agents, achieving human-level accuracy through memory-augmented reasoning and a new benchmark.

Contribution

It introduces a universal, memory-augmented reasoning framework for LLM evaluators and presents ASSEBench, a comprehensive benchmark for safety and security assessment.

Findings

AgentAuditor outperforms existing evaluators in safety/security detection.

Achieves human-level accuracy in safety and security evaluation.

Sets new state-of-the-art performance on ASSEBench.

Abstract

Despite the rapid advancement of LLM-based agents, the reliable evaluation of their safety and security remains a significant challenge. Existing rule-based or LLM-based evaluators often miss dangers in agents' step-by-step actions, overlook subtle meanings, fail to see how small issues compound, and get confused by unclear safety or security rules. To overcome this evaluation crisis, we introduce AgentAuditor, a universal, training-free, memory-augmented reasoning framework that empowers LLM evaluators to emulate human expert evaluators. AgentAuditor constructs an experiential memory by having an LLM adaptively extract structured semantic features (e.g., scenario, risk, behavior) and generate associated chain-of-thought reasoning traces for past interactions. A multi-stage, context-aware retrieval-augmented generation process then dynamically retrieves the most relevant reasoning experiences to guide the LLM evaluator's assessment of new cases. Moreover, we developed ASSEBench, the first benchmark designed to check how well LLM-based evaluators can spot both safety risks and security threats. ASSEBench comprises 2293 meticulously annotated interaction records, covering 15 risk types across 29 application scenarios. A key feature of ASSEBench is its nuanced approach to ambiguous risk situations, employing "Strict" and "Lenient" judgment standards. Experiments demonstrate that AgentAuditor not only consistently improves the evaluation performance of LLMs across all benchmarks but also sets a new state-of-the-art in LLM-as-a-judge for agent safety and security, achieving human-level accuracy. Our work is openly accessible at https://github.com/Astarojth/AgentAuditor.