Analyzing Information Sharing and Coordination in Multi-Agent Planning

TL;DR

This paper explores how structured information sharing and orchestration improve multi-agent LLM systems for complex travel planning, significantly boosting accuracy and coordination.

Contribution

It introduces a novel multi-agent system with a notebook for info sharing and an orchestrator for coordination, demonstrating substantial performance improvements.

Findings

Notebook reduces hallucinated errors by 18%

Orchestrator improves focus and reduces errors by 13.5%

Combined mechanisms achieve 25% pass rate on TravelPlanner benchmark

Abstract

Multi-agent systems (MASs) have pushed the boundaries of large language model (LLM) agents in domains such as web research and software engineering. However, long-horizon, multi-constraint planning tasks involve conditioning on detailed information and satisfying complex interdependent constraints, which can pose a challenge for these systems. In this study, we construct an LLM-based MAS for a travel planning task which is representative of these challenges. We evaluate the impact of a notebook to facilitate information sharing, and evaluate an orchestrator agent to improve coordination in free form conversation between agents. We find that the notebook reduces errors due to hallucinated details by 18%, while an orchestrator directs the MAS to focus on and further reduce errors by up to 13.5% within focused sub-areas. Combining both mechanisms achieves a 25% final pass rate on the TravelPlanner benchmark, a 17.5% absolute improvement over the single-agent baseline's 7.5% pass rate. These results highlight the potential of structured information sharing and reflective orchestration as key components in MASs for long horizon planning with LLMs.