Autonomous Adversary: Red-Teaming in the age of LLM

TL;DR

This paper explores the use of Language Model Agents (LMAs) in red-team cyber operations, benchmarking their performance in simulated lateral-movement scenarios and analyzing their strengths and limitations.

Contribution

It introduces a framework for evaluating LMAs in adversary emulation, formalizes scenarios with validation, and compares different operational modalities for autonomous cyber attack planning.

Findings

Expert-defined plans outperform autonomous modes in task completion.

Failure causes include brittle commands and environmental instability.

LMAs show potential but face significant reliability challenges.

Abstract

Language Model Agents (LMAs) are emerging as a powerful primitive for augmenting red-team operations. They can support attack planning, adversary emulation, and the orchestration of multi-step activity such as lateral movement, a core enabling capability of advanced persistent threat (APT) campaigns. Using frameworks such as MITRE ATT&CK, we analyze where these agents intersect with core offensive functions and assess current strengths and limitations of LMAs with an emphasis on governance and realistic evaluation. We benchmark LMAs across two lateral-movement scenarios in a controlled adversary-emulation environment, where LMAs interact with instrumented cyber agents, observe execution artifacts, and iteratively adapt based on environmental feedback. Each scenario is formalized as an ordered task chain with explicit validation predicates, leveraging an LLM-as-a-Judge paradigm to ensure deterministic outcome verification. We compare three operational modalities: fully autonomous execution, self-scaffolded planning, and expert-defined action plans. Preliminary findings indicate that expert-defined action plans yield higher task-completion rates relative to other operational modes. However, failure remains frequent across all modalities, largely attributable to brittle command invocation, environmental and deployment instability, and recurring errors in credential management and state handling.