MASAI: Modular Architecture for Software-engineering AI Agents

TL;DR

MASAI introduces a modular, multi-agent architecture for software engineering tasks using LLMs, improving problem-solving efficiency and achieving high resolution rates on challenging datasets.

Contribution

This paper presents MASAI, a novel modular architecture for AI agents in software engineering, enabling diverse strategies and source gathering, with demonstrated superior performance.

Findings

Achieved 28.33% resolution rate on SWE-bench Lite dataset.

Enabled problem-solving with multiple specialized sub-agents.

Outperformed other agentic methods in evaluations.

Abstract

A common method to solve complex problems in software engineering, is to divide the problem into multiple sub-problems. Inspired by this, we propose a Modular Architecture for Software-engineering AI (MASAI) agents, where different LLM-powered sub-agents are instantiated with well-defined objectives and strategies tuned to achieve those objectives. Our modular architecture offers several advantages: (1) employing and tuning different problem-solving strategies across sub-agents, (2) enabling sub-agents to gather information from different sources scattered throughout a repository, and (3) avoiding unnecessarily long trajectories which inflate costs and add extraneous context. MASAI enabled us to achieve the highest performance (28.33% resolution rate) on the popular and highly challenging SWE-bench Lite dataset consisting of 300 GitHub issues from 11 Python repositories. We conduct a comprehensive evaluation of MASAI relative to other agentic methods and analyze the effects of our design decisions and their contribution to the success of MASAI.