Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents

TL;DR

This paper introduces MT-AgentRisk, a benchmark for multi-turn safety evaluation of tool-using agents, and proposes ToolShield, a self-exploration defense that significantly reduces safety risks in multi-turn interactions.

Contribution

It presents the first multi-turn safety benchmark and a novel, training-free defense method for improving safety in tool-using language agents.

Findings

Multi-turn safety risks are significantly higher, with a 16% increase in Attack Success Rate.

ToolShield reduces safety failure rates by 30% in multi-turn scenarios.

The benchmark enables systematic safety evaluation in realistic multi-turn, tool-using settings.

Abstract

LLM-based agents are becoming increasingly capable, yet their safety lags behind. This creates a gap between what agents can do and should do. This gap widens as agents engage in multi-turn interactions and employ diverse tools, introducing new risks overlooked by existing benchmarks. To systematically scale safety testing into multi-turn, tool-realistic settings, we propose a principled taxonomy that transforms single-turn harmful tasks into multi-turn attack sequences. Using this taxonomy, we construct MT-AgentRisk (Multi-Turn Agent Risk Benchmark), the first benchmark to evaluate multi-turn tool-using agent safety. Our experiments reveal substantial safety degradation: the Attack Success Rate (ASR) increases by 16% on average across open and closed models in multi-turn settings. To close this gap, we propose ToolShield, a training-free, tool-agnostic, self-exploration defense: when encountering a new tool, the agent autonomously generates test cases, executes them to observe downstream effects, and distills safety experiences for deployment. Experiments show that ToolShield effectively reduces ASR by 30% on average in multi-turn interactions. Our code is available at https://github.com/CHATS-lab/ToolShield.