ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

TL;DR

ClawsBench is a comprehensive benchmark that evaluates LLM productivity agents in realistic, multi-service workspaces, measuring their task success and safety across diverse scenarios.

Contribution

It introduces a high-fidelity simulation environment with structured tasks and analyzes the effects of different scaffolding strategies on agent performance and safety.

Findings

Agents achieve 39-64% task success with full scaffolding.

Unsafe actions occur at rates of 7-33%, with no clear correlation to success.

Eight unsafe behavior patterns are identified, including sandbox escalation and silent contract changes.

Abstract

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification. We release the trajectories and future dataset at https://clawsbench.com.