Safety Training Persists Through Helpfulness Optimization in LLM Agents

TL;DR

This paper investigates how safety training in large language model (LLM) agents persists through helpfulness optimization, revealing that safety and helpfulness trade-offs follow a linear Pareto frontier and emphasizing the need for better understanding of post-training dynamics.

Contribution

It demonstrates that safety training persists through helpfulness training and that all training configurations align along a linear Pareto frontier, challenging assumptions about optimal combined training strategies.

Findings

Safety training persists through helpfulness training.

Training configurations align along a linear Pareto frontier.

Post-training on both metrics results in points on the frontier, not an optimal trade-off.

Abstract

Safety post-training has been studied extensively in single-step "chat" settings where safety typically refers to refusing harmful requests. We study an "agentic" (i.e., multi-step, tool-use) setting where safety refers to harmful actions directly taken by the LLM. We compare the effects of running direct preference optimization (DPO) on safety or helpfulness alone vs both metrics sequentially. As expected, training on one metric alone results in an extreme point along this frontier. However, unlike prior work, we find that safety training persists through subsequent helpfulness training. We also find that all training configurations end up near a linear Pareto frontier with $R^2 = 0.77$. Even post-training on both metrics simultaneously simply results in another point on the frontier rather than finding a "best of both worlds" strategy, despite the presence of such strategies in our DPO dataset. Overall, our findings underscore the need for better understanding of post-training dynamics.