SafePro: Evaluating the Safety of Professional-Level AI Agents

TL;DR

SafePro introduces a comprehensive benchmark for evaluating the safety of AI agents performing complex professional tasks, revealing significant safety vulnerabilities and the need for improved safety mechanisms in advanced AI systems.

Contribution

The paper presents SafePro, a novel benchmark dataset for assessing safety in professional AI agents, addressing a gap in existing safety evaluations for complex, real-world tasks.

Findings

State-of-the-art models show safety vulnerabilities in professional tasks.

Models exhibit insufficient safety judgment and alignment.

Safety mitigation strategies improve agent safety.

Abstract

Large language model-based agents are rapidly evolving from simple conversational assistants into autonomous systems capable of performing complex, professional-level tasks in various domains. While these advancements promise significant productivity gains, they also introduce critical safety risks that remain under-explored. Existing safety evaluations primarily focus on simple, daily assistance tasks, failing to capture the intricate decision-making processes and potential consequences of misaligned behaviors in professional settings. To address this gap, we introduce \textbf{SafePro}, a comprehensive benchmark designed to evaluate the safety alignment of AI agents performing professional activities. SafePro features a dataset of high-complexity tasks across diverse professional domains with safety risks, developed through a rigorous iterative creation and review process. Our evaluation of state-of-the-art AI models reveals significant safety vulnerabilities and uncovers new unsafe behaviors in professional contexts. We further show that these models exhibit both insufficient safety judgment and weak safety alignment when executing complex professional tasks. In addition, we investigate safety mitigation strategies for improving agent safety in these scenarios and observe encouraging improvements. Together, our findings highlight the urgent need for robust safety mechanisms tailored to the next generation of professional AI agents.