Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development

TL;DR

This paper presents an empirical study on optimizing role-based multi-agent systems using textual feedback, improving performance in complex software development tasks through a two-step prompt optimization pipeline and analyzing various optimization strategies.

Contribution

It introduces a novel two-step prompt optimization pipeline utilizing textual feedback and compares different optimization strategies for enhancing multi-agent system performance.

Findings

Optimization improves multi-agent system performance on software tasks.

Group optimization strategies outperform individual ones.

Multi-pass prompting yields better results than one-pass.

Abstract

We have seen remarkable progress in large language models (LLMs) empowered multi-agent systems solving complex tasks necessitating cooperation among experts with diverse skills. However, optimizing LLM-based multi-agent systems remains challenging. In this work, we perform an empirical case study on group optimization of role-based multi-agent systems utilizing natural language feedback for challenging software development tasks under various evaluation dimensions. We propose a two-step agent prompts optimization pipeline: identifying underperforming agents with their failure explanations utilizing textual feedback and then optimizing system prompts of identified agents utilizing failure explanations. We then study the impact of various optimization settings on system performance with two comparison groups: online against offline optimization and individual against group optimization. For group optimization, we study two prompting strategies: one-pass and multi-pass prompting optimizations. Overall, we demonstrate the effectiveness of our optimization method for role-based multi-agent systems tackling software development tasks evaluated on diverse evaluation dimensions, and we investigate the impact of diverse optimization settings on group behaviors of the multi-agent systems to provide practical insights for future development.