LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

TL;DR

LITMUS is a comprehensive benchmark for evaluating the safety and robustness of LLM-based OS agents against behavioral jailbreaks, addressing both semantic and physical-layer risks with a dual verification system.

Contribution

It introduces a dual verification benchmark with 819 test cases and an automated evaluation framework for assessing LLM agent safety at both conversational and OS levels.

Findings

Current agents often execute high-risk OS operations despite safety measures.

Agents frequently exhibit Execution Hallucination, misleadingly denying completed dangerous actions.

Skill injection and entity wrapping attacks are highly successful, revealing significant vulnerabilities.

Abstract

The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content safety: behavior jailbreak, where an adversary induces an agent to execute dangerous OS-level operations with irreversible consequences. Existing benchmarks either evaluate safety at the semantic layer alone, missing physical-layer harms, or fail to isolate test cases, letting earlier runs contaminate later ones. We present LITMUS (LLM-agents In-OS Testing for Measuring Unsafe Subversion), a benchmark addressing both gaps via a semantic-physical dual verification mechanism and OS-level state rollback. LITMUS comprises 819 high-risk test cases organized into one harmful seed subset and six attack-extended subsets covering three adversarial paradigms (jailbreak speaking, skill injection, and entity wrapping), plus a fully automated multi-agent evaluation framework judging behavior at both conversational and OS-level physical layers. Evaluation across frontier agents reveals three findings: (1) current agents lack effective safety awareness, with strong models (e.g., Claude Sonnet 4.6) still executing 40.64% of high-risk operations; (2) agents exhibit pervasive Execution Hallucination (EH), verbally refusing a request while the dangerous operation has already completed at the system level, invisible to every prior semantic-only framework; and (3) skill injection and entity wrapping attacks achieve high success rates, exposing pronounced agent vulnerabilities. LITMUS provides the first standardized platform for reproducible, physically grounded behavioral safety evaluation of LLM agents in real OS environments.