MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

TL;DR

MapCoder introduces a multi-agent prompting framework that emulates human program synthesis stages, significantly improving code generation performance on multiple benchmarks and achieving state-of-the-art results.

Contribution

The paper presents a novel multi-agent prompting approach for code generation, replicating human-like synthesis stages to enhance LLM performance.

Findings

Achieves state-of-the-art pass@1 on HumanEval (93.9%)

Outperforms previous methods on MBPP, APPS, CodeContests, and xCodeEval

Demonstrates robustness across programming languages and problem difficulties.

Abstract

Code synthesis, which requires a deep understanding of complex natural language problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge. While large language models (LLMs) demonstrate impressive proficiency in natural language processing, their performance in code generation tasks remains limited. In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers. Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging. After conducting thorough experiments, with multiple LLM ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks, MapCoder showcases remarkable code generation capabilities, achieving new state-of-the-art results (pass@1) on HumanEval (93.9%), MBPP (83.1%), APPS (22.0%), CodeContests (28.5%), and xCodeEval (45.3%). Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties. We open-source our framework at https://github.com/Md-Ashraful-Pramanik/MapCoder.