Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration

TL;DR

This paper demonstrates that sharing context between two commercial AI tools, PRD AI and GitHub Copilot, enhances code suggestion accuracy and developer success rates in real-world software development scenarios.

Contribution

It provides a novel real-world experiment showing how integrating commercial AI tools in a multi-agent setup improves software development outcomes.

Findings

GitHub Copilot's code suggestions improved by 13.8%

Developer task success rate increased by 24.5%

First real-world example of commercial AI tools collaboration

Abstract

In recent years, with the rapid advancement of large language models (LLMs), multi-agent systems have become increasingly more capable of practical application. At the same time, the software development industry has had a number of new AI-powered tools developed that improve the software development lifecycle (SDLC). Academically, much attention has been paid to the role of multi-agent systems to the SDLC. And, while single-agent systems have frequently been examined in real-world applications, we have seen comparatively few real-world examples of publicly available commercial tools working together in a multi-agent system with measurable improvements. In this experiment we test context sharing between Crowdbotics PRD AI, a tool for generating software requirements using AI, and GitHub Copilot, an AI pair-programming tool. By sharing business requirements from PRD AI, we improve the code suggestion capabilities of GitHub Copilot by 13.8% and developer task success rate by 24.5% -- demonstrating a real-world example of commercially-available AI systems working together with improved outcomes.