Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions

TL;DR

This paper introduces a multi-dimensional evasion framework and benchmark to evaluate vulnerabilities in LLM-based autonomous agents, revealing significant security risks overlooked by current defenses.

Contribution

It presents a novel multi-dimensional evasion framework and a comprehensive benchmark, exposing systemic vulnerabilities in autonomous agents against sophisticated attack vectors.

Findings

Evasion framework increases risk trigger rate from 28.3% to 52.6%.

Systemic vulnerabilities exist in current autonomous agent architectures.

Existing defenses are insufficient against multi-turn, multi-vector attacks.

Abstract

As autonomous agents (e.g., OpenClaw) increasingly operate with deep system-level privileges to execute complex tasks, they introduce severe, unmitigated security risks. Current vulnerability analyses overwhelmingly focus on single-turn, stateless behaviors, overlooking the expanded attack surface inherent in stateful, multi-turn interactions and dynamic tool invocations. In this paper, we propose a novel, multi-dimensional evasion framework targeting LLM-based agent systems. We introduce three stealthy attack vectors: (1) Temporal evasion, which fragments malicious payloads across sequential interaction turns; (2) Spatial evasion, which conceals payloads within complex external artifacts that evade standard LLM parsing mechanisms; and (3) Semantic evasion, which obscures malicious intents beneath benign contextual noise. To systematically quantify these threats, we construct A3S-Bench, a comprehensive benchmark comprising 2,254 real-world agent execution trajectories. Evaluating a standard agent framework separately integrated with 10 mainstream LLM backbones against 20 practical threat scenarios, we demonstrate that our evasion framework elevates the average risk trigger rate from a 28.3\% baseline to 52.6\%. These findings reveal systemic, architecture-level vulnerabilities in current autonomous agent systems that existing defenses fail to address, highlighting an urgent need for defense mechanisms tailored to the unique threats.