SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors

TL;DR

SLEIGHT-Bench is a comprehensive benchmark testing the ability of agent monitors to detect covert harmful behaviors in autonomous coding agents, revealing significant evasion vulnerabilities.

Contribution

The paper introduces SLEIGHT-Bench, a new benchmark dataset with diverse attack strategies, and evaluates monitor effectiveness, highlighting key evasion tactics and potential improvements.

Findings

20 of 40 attacks evade current monitors at 1% false-positive rate

Overall detection rate is 32% across all attacks

Targeted prompts improve monitor detection performance

Abstract

Since autonomous coding agents generate complex behaviors at high-volume, we may want to use other LLMs to monitor actions to reduce the risk from dangerous misaligned behavior. To better understand the limitations of such monitors against the diverse attack strategies that a coding agent could use, we present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories, each showing a coding agent covertly pursuing a harmful objective (e.g. weight exfiltration, credential theft, unmonitored rogue deployment). We find that 20 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 32%. We identify several evasion strategies such as exploiting model priors, user instruction ambiguity, and state manipulation, which facilitate creating attacks that evade frontier monitors. We also elicit stronger monitor performance using coding agents as monitors versus regular prompted monitors, and for some evasion strategies show improved catch rates with targeted monitor prompts. Our dataset and evaluation framework are available at https://github.com/safety-research/sleight-bench and https://huggingface.co/datasets/sleightbench/SLEIGHT-Bench.