Understanding Multi-Agent LLM Frameworks: A Unified Benchmark and Experimental Analysis

TL;DR

This paper introduces a unified benchmark and analysis for multi-agent LLM frameworks, revealing how architectural choices significantly affect performance, accuracy, and scalability, and providing guidance for future design.

Contribution

It presents an architectural taxonomy and MAFBench, a standardized evaluation suite, enabling systematic comparison and analysis of multi-agent LLM frameworks.

Findings

Framework design impacts latency by over 100x

Planning accuracy can decrease by up to 30%

Coordination success drops from over 90% to below 30%

Abstract

Multi-agent LLM frameworks are widely used to accelerate the development of agent systems powered by large language models (LLMs). These frameworks impose distinct architectural structures that govern how agents interact, store information, and coordinate tasks. However, their impact on system performance remains poorly understood. This gap is critical, as architectural choices alone can induce order-of-magnitude differences in latency and throughput, as well as substantial variation in accuracy and scalability. Addressing this challenge requires (i) jointly evaluating multiple capabilities, such as orchestration overhead, memory behavior, planning, specialization, and coordination, and (ii) conducting these evaluations under controlled, framework-level conditions to isolate architectural effects. Existing benchmarks focus on individual capabilities and lack standardized framework-level evaluation. We address these limitations by (i) introducing an architectural taxonomy for systematically comparing multi-agent LLM frameworks along fundamental dimensions, and (ii) developing MAFBench, a unified evaluation suite that integrates existing benchmarks under a standardized execution pipeline. Using MAFBench, we conduct a controlled empirical study across several widely used frameworks. Our results show that framework-level design choices alone can increase latency by over 100x, reduce planning accuracy by up to 30%, and lower coordination success from above 90% to below 30%. Finally, we translate our findings into concrete architectural design principles and framework selection guidance, and outline promising future research directions.