BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents

TL;DR

This paper introduces BadAgent, a backdoor attack method on LLM-based agents that can manipulate their behavior at test time, revealing significant security vulnerabilities in current fine-tuning practices.

Contribution

It is the first to demonstrate backdoor attacks on LLM agents, showing their robustness and potential for malicious manipulation using trigger-based inputs.

Findings

Backdoor attacks remain effective after fine-tuning on trustworthy data.

The attack can manipulate agents to perform harmful operations.

This work highlights security risks in deploying LLM agents from untrusted sources.

Abstract

With the prosperity of large language models (LLMs), powerful LLM-based intelligent agents have been developed to provide customized services with a set of user-defined tools. State-of-the-art methods for constructing LLM agents adopt trained LLMs and further fine-tune them on data for the agent task. However, we show that such methods are vulnerable to our proposed backdoor attacks named BadAgent on various agent tasks, where a backdoor can be embedded by fine-tuning on the backdoor data. At test time, the attacker can manipulate the deployed LLM agents to execute harmful operations by showing the trigger in the agent input or environment. To our surprise, our proposed attack methods are extremely robust even after fine-tuning on trustworthy data. Though backdoor attacks have been studied extensively in natural language processing, to the best of our knowledge, we could be the first to study them on LLM agents that are more dangerous due to the permission to use external tools. Our work demonstrates the clear risk of constructing LLM agents based on untrusted LLMs or data. Our code is public at https://github.com/DPamK/BadAgent