GEMMAS: Graph-based Evaluation Metrics for Multi Agent Systems

TL;DR

GEMMAS introduces a graph-based framework with process-level metrics to evaluate and analyze the internal collaboration quality of multi-agent systems, revealing insights beyond final output accuracy.

Contribution

It proposes novel process-level metrics, IDS and UPR, for analyzing agent interactions and collaboration efficiency in multi-agent systems.

Findings

Significant variation in collaboration quality despite similar accuracy.

Outcome metrics alone are insufficient for comprehensive evaluation.

Process-level diagnostics can guide more interpretable and resource-efficient systems.

Abstract

Multi-agent systems built on language models have shown strong performance on collaborative reasoning tasks. However, existing evaluations focus only on the correctness of the final output, overlooking how inefficient communication and poor coordination contribute to redundant reasoning and higher computational costs. We introduce GEMMAS, a graph-based evaluation framework that analyzes the internal collaboration process by modeling agent interactions as a directed acyclic graph. To capture collaboration quality, we propose two process-level metrics: Information Diversity Score (IDS) to measure semantic variation in inter-agent messages, and Unnecessary Path Ratio (UPR) to quantify redundant reasoning paths. We evaluate GEMMAS across five benchmarks and highlight results on GSM8K, where systems with only a 2.1% difference in accuracy differ by 12.8% in IDS and 80% in UPR, revealing substantial variation in internal collaboration. These findings demonstrate that outcome-only metrics are insufficient for evaluating multi-agent performance and highlight the importance of process-level diagnostics in designing more interpretable and resource-efficient collaborative AI systems.