LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review

TL;DR

This literature review synthesizes academic and industrial research on LLM-based multi-agent systems for code generation, highlighting motivations, models, challenges, solutions, and future directions.

Contribution

It is the first comprehensive multi-vocal review combining peer-reviewed and grey literature on this topic, providing structured insights and identifying research gaps.

Findings

Classified nine categories of reasons for adopting multi-agent systems

Analyzed models and benchmarks to overview LLM configurations and evaluation practices

Synthesized challenges and solutions into six main categories and 26 subcategories

Abstract

Large Language Models (LLMs) have enabled multi-agent systems to perform autonomous code generation for complex tasks. Despite the recent growth in research and industrial applications in this area, there is little work on synthesizing evidence from both academic and industrial sources to capture the current state of research on LLM-based multi-agent systems for code generation. To this end, we conducted a Multi-Vocal Literature Review (MLR), combining insights from both academia and industry, including peer-reviewed studies and grey literature. The aim of this study is to systematically synthesize and analyze existing knowledge on LLM-based multi-agent systems for code generation. Specifically, the review examines the motivations for their use, employed benchmarks and models, key challenges, proposed solutions, and potential directions for future research. We selected and reviewed 114 studies, and the key findings are: 1) the identified reasons for adopting multi-agent systems for code generation were classified into nine categories; 2) the models and evaluation benchmarks utilized across the studies were systematically analyzed to provide a structured overview of commonly adopted LLM configurations and assessment practices; 3) the reported challenges and corresponding solutions were synthesized into six main categories and 26 subcategories; and 4) future research directions were identified and organized into six main categories and 18 subcategories. The results of this MLR will assist researchers and practitioners in pursuing further studies and supporting the real-world adoption of multi-agent systems in industrial settings.