Evil Geniuses: Delving into the Safety of LLM-based Agents

TL;DR

This paper investigates the safety risks of LLM-based agents by proposing attack strategies that reveal their vulnerabilities, demonstrating that these agents are less robust and more prone to harmful behaviors than standalone LLMs.

Contribution

It introduces Evil Geniuses, an autonomous attack method that tests the safety of LLM-based agents across various roles and attack levels, highlighting their safety challenges.

Findings

Agents are less robust and more vulnerable to attacks.

LLM-based agents can generate more harmful and stealthier content.

The proposed attack methods achieve high success rates in evaluations.

Abstract

Rapid advancements in large language models (LLMs) have revitalized in LLM-based agents, exhibiting impressive human-like behaviors and cooperative capabilities in various scenarios. However, these agents also bring some exclusive risks, stemming from the complexity of interaction environments and the usability of tools. This paper delves into the safety of LLM-based agents from three perspectives: agent quantity, role definition, and attack level. Specifically, we initially propose to employ a template-based attack strategy on LLM-based agents to find the influence of agent quantity. In addition, to address interaction environment and role specificity issues, we introduce Evil Geniuses (EG), an effective attack method that autonomously generates prompts related to the original role to examine the impact across various role definitions and attack levels. EG leverages Red-Blue exercises, significantly improving the generated prompt aggressiveness and similarity to original roles. Our evaluations on CAMEL, Metagpt and ChatDev based on GPT-3.5 and GPT-4, demonstrate high success rates. Extensive evaluation and discussion reveal that these agents are less robust, prone to more harmful behaviors, and capable of generating stealthier content than LLMs, highlighting significant safety challenges and guiding future research. Our code is available at https://github.com/T1aNS1R/Evil-Geniuses.