Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development

TL;DR

This paper examines the vulnerabilities of multi-agent AI systems in software development, especially to code injection attacks, and proposes a security analysis agent to improve resilience, highlighting the trade-offs between efficiency and security.

Contribution

It introduces a threat model for multi-agent systems in software engineering and demonstrates how adding a security analysis agent enhances security against code injection attacks.

Findings

Coder-reviewer-tester architecture is more resilient than other architectures.

Adding a security analysis agent improves security and maintains efficiency.

Advanced code injection can significantly increase attack success rate.

Abstract

Agentic AI and Multi-Agent Systems are poised to dominate industry and society imminently. Powered by goal-driven autonomy, they represent a powerful form of generative AI, marking a transition from reactive content generation into proactive multitasking capabilities. As an exemplar, we propose an architecture of a multi-agent system for the implementation phase of the software engineering process. We also present a comprehensive threat model for the proposed system. We demonstrate that while such systems can generate code quite accurately, they are vulnerable to attacks, including code injection. Due to their autonomous design and lack of humans in the loop, these systems cannot identify and respond to attacks by themselves. This paper analyzes the vulnerability of multi-agent systems and concludes that the coder-reviewer-tester architecture is more resilient than both the coder and coder-tester architectures, but is less efficient at writing code. We find that by adding a security analysis agent, we mitigate the loss in efficiency while achieving even better resiliency. We conclude by demonstrating that the security analysis agent is vulnerable to advanced code injection attacks, showing that embedding poisonous few-shot examples in the injected code can increase the attack success rate from 0% to 71.95%.