AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models

TL;DR

AgentAlign introduces a novel safety alignment framework for large language models, using behavior chains in simulated environments to improve safety without sacrificing helpfulness, addressing malicious use risks.

Contribution

The paper presents a new method leveraging behavior chains for safety alignment in LLMs, enhancing safety while maintaining utility, and provides open-source datasets and code.

Findings

Safety improved from 35.8% to 79.5%

Enhanced safety with minimal impact on helpfulness

Outperforms existing prompting methods

Abstract

The acquisition of agentic capabilities has transformed LLMs from "knowledge providers" to "action executors", a trend that while expanding LLMs' capability boundaries, significantly increases their susceptibility to malicious use. Previous work has shown that current LLM-based agents execute numerous malicious tasks even without being attacked, indicating a deficiency in agentic use safety alignment during the post-training phase. To address this gap, we propose AgentAlign, a novel framework that leverages abstract behavior chains as a medium for safety alignment data synthesis. By instantiating these behavior chains in simulated environments with diverse tool instances, our framework enables the generation of highly authentic and executable instructions while capturing complex multi-step dynamics. The framework further ensures model utility by proportionally synthesizing benign instructions through non-malicious interpretations of behavior chains, precisely calibrating the boundary between helpfulness and harmlessness. Evaluation results on AgentHarm demonstrate that fine-tuning three families of open-source models using our method substantially improves their safety (35.8% to 79.5% improvement) while minimally impacting or even positively enhancing their helpfulness, outperforming various prompting methods. The dataset and code have both been open-sourced.