Agent Safety Alignment via Reinforcement Learning

TL;DR

This paper introduces a unified safety-alignment framework for tool-using autonomous LLM agents, addressing threats from both user prompts and malicious tools through structured reasoning and sandboxed reinforcement learning.

Contribution

It presents the first comprehensive safety-alignment approach for tool-using agents, integrating a tri-modal taxonomy and a sandbox environment for improved security and utility.

Findings

Enhanced resistance to security threats in agents

Maintained strong performance on benign tasks

Joint optimization of safety and effectiveness

Abstract

The emergence of autonomous Large Language Model (LLM) agents capable of tool usage has introduced new safety risks that go beyond traditional conversational misuse. These agents, empowered to execute external functions, are vulnerable to both user-initiated threats (e.g., adversarial prompts) and tool-initiated threats (e.g., malicious outputs from compromised tools). In this paper, we propose the first unified safety-alignment framework for tool-using agents, enabling models to handle both channels of threat via structured reasoning and sandboxed reinforcement learning. We introduce a tri-modal taxonomy, including benign, malicious, and sensitive for both user prompts and tool responses, and define a policy-driven decision model. Our framework employs a custom-designed sandbox environment that simulates real-world tool execution and allows fine-grained reward shaping. Through extensive evaluations on public and self-built benchmarks, including Agent SafetyBench, InjecAgent, and BFCL, we demonstrate that our safety-aligned agents significantly improve resistance to security threats while preserving strong utility on benign tasks. Our results show that safety and effectiveness can be jointly optimized, laying the groundwork for trustworthy deployment of autonomous LLM agents.