RRTL: Red Teaming Reasoning Large Language Models in Tool Learning

TL;DR

This paper introduces RRTL, a red teaming method to evaluate the safety of reasoning large language models (RLLMs) in tool learning, revealing safety strengths and vulnerabilities across models.

Contribution

It presents a novel red teaming approach with two strategies to assess RLLMs' safety and uncovers key safety challenges and disparities among models.

Findings

RLLMs generally outperform traditional LLMs in safety.

Substantial safety disparities exist across models.

Deceptive risks and multilingual vulnerabilities are prevalent.

Abstract

While tool learning significantly enhances the capabilities of large language models (LLMs), it also introduces substantial security risks. Prior research has revealed various vulnerabilities in traditional LLMs during tool learning. However, the safety of newly emerging reasoning LLMs (RLLMs), such as DeepSeek-R1, in the context of tool learning remains underexplored. To bridge this gap, we propose RRTL, a red teaming approach specifically designed to evaluate RLLMs in tool learning. It integrates two novel strategies: (1) the identification of deceptive threats, which evaluates the model's behavior in concealing the usage of unsafe tools and their potential risks; and (2) the use of Chain-of-Thought (CoT) prompting to force tool invocation. Our approach also includes a benchmark for traditional LLMs. We conduct a comprehensive evaluation on seven mainstream RLLMs and uncover three key findings: (1) RLLMs generally achieve stronger safety performance than traditional LLMs, yet substantial safety disparities persist across models; (2) RLLMs can pose serious deceptive risks by frequently failing to disclose tool usage and to warn users of potential tool output risks; (3) CoT prompting reveals multi-lingual safety vulnerabilities in RLLMs. Our work provides important insights into enhancing the security of RLLMs in tool learning.