Empirical Comparison of Agent Communication Protocols for Task Orchestration

TL;DR

This paper develops a systematic benchmark to compare communication protocols for LLM agent task orchestration, focusing on tool integration, multi-agent delegation, and hybrid architectures.

Contribution

It introduces a pilot benchmark to evaluate various communication protocols for LLM agents across multiple metrics and complexity levels.

Findings

Benchmark reveals trade-offs in response time and cost among protocols.

Hybrid architectures show advantages in error recovery.

Tool integration impacts context window consumption significantly.

Abstract

Context. The problem of comparative evaluation of communication protocols for task orchestration by large language model (LLM) agents is considered. The object of study is the process of interaction between LLM agents and external tools, as well as between autonomous LLM agents, during task orchestration. Objective. The goal of this work is to develop a systematic pilot benchmark comparing tool integration, multi-agent dele-gation, and hybrid architectures for standardized queries at three levels of complexity, and to quantify the advantages and disadvantages in terms of response time, context window consumption, cost, error recovery, and implementation complexity.