Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities

TL;DR

This paper uncovers security vulnerabilities in LLM-based multi-agent systems, demonstrating how manipulated knowledge can spread covertly and persist, posing significant risks that require new defense strategies.

Contribution

It introduces a novel two-stage attack method exploiting LLM vulnerabilities to spread manipulated knowledge without prompt tampering, supported by extensive experiments.

Findings

Successful manipulation of knowledge spread without degrading agent performance

Manipulated knowledge persists through retrieval-augmented frameworks

Highlights need for robust defenses like guardian agents and fact-checkers

Abstract

The rapid adoption of large language models (LLMs) in multi-agent systems has highlighted their impressive capabilities in various applications, such as collaborative problem-solving and autonomous negotiation. However, the security implications of these LLM-based multi-agent systems have not been thoroughly investigated, particularly concerning the spread of manipulated knowledge. In this paper, we investigate this critical issue by constructing a detailed threat model and a comprehensive simulation environment that mirrors real-world multi-agent deployments in a trusted platform. Subsequently, we propose a novel two-stage attack method involving Persuasiveness Injection and Manipulated Knowledge Injection to systematically explore the potential for manipulated knowledge (i.e., counterfactual and toxic knowledge) spread without explicit prompt manipulation. Our method leverages the inherent vulnerabilities of LLMs in handling world knowledge, which can be exploited by attackers to unconsciously spread fabricated information. Through extensive experiments, we demonstrate that our attack method can successfully induce LLM-based agents to spread both counterfactual and toxic knowledge without degrading their foundational capabilities during agent communication. Furthermore, we show that these manipulations can persist through popular retrieval-augmented generation frameworks, where several benign agents store and retrieve manipulated chat histories for future interactions. This persistence indicates that even after the interaction has ended, the benign agents may continue to be influenced by manipulated knowledge. Our findings reveal significant security risks in LLM-based multi-agent systems, emphasizing the imperative need for robust defenses against manipulated knowledge spread, such as introducing ``guardian'' agents and advanced fact-checking tools.