Tool Unlearning for Tool-Augmented LLMs

TL;DR

This paper introduces ToolDelete, a novel method for unlearning specific tools from tool-augmented large language models, addressing security, privacy, and deprecation issues while maintaining overall model performance.

Contribution

The paper presents the first approach for unlearning tools in LLMs, including a new evaluation metric and extensive experiments demonstrating effectiveness.

Findings

ToolDelete effectively unlearns selected tools.

It preserves knowledge of non-deleted tools.

Maintains performance on general tasks.

Abstract

Tool-augmented large language models (LLMs) are often trained on datasets of query-response pairs, which embed the ability to use tools or APIs directly into the parametric knowledge of LLMs. Tool-augmented LLMs need the ability to forget learned tools due to security vulnerabilities, privacy regulations, or tool deprecations. However, ``tool unlearning'' has not been investigated in unlearning literature. We introduce this novel task, which requires addressing distinct challenges compared to traditional unlearning: knowledge removal rather than forgetting individual samples, the high cost of optimizing LLMs, and the need for principled evaluation metrics. To bridge these gaps, we propose ToolDelete, the first approach for unlearning tools from tool-augmented LLMs. It implements three key properties to address the above challenges for effective tool unlearning and introduces a new membership inference attack (MIA) model for effective evaluation. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that ToolDelete effectively unlearns randomly selected tools, while preserving the LLM's knowledge on non-deleted tools and maintaining performance on general tasks.