Efficient Mixture-of-Agents Serving via Tree-Structured Routing, Adaptive Pruning, and Dependency-Aware Prefill-Decode Overlap

TL;DR

This paper introduces a novel MoA serving approach that reduces latency by restructuring agent communication, adaptively skipping computations, and overlapping processing, achieving up to 90% latency reduction with maintained accuracy.

Contribution

It proposes a hierarchical tree topology, adaptive runtime skipping, and overlapping execution techniques to improve MoA serving efficiency and latency.

Findings

Up to 90% reduction in end-to-end latency.

Maintains accuracy within ±1% of dense MoA baselines.

Improves hardware utilization and inference speed.

Abstract

Mixture-of-Agents (MoA) inference can suffer from dense inter-agent communication and low hardware utilization, which jointly inflate serving latency. We present a serving design that targets these bottlenecks through an algorithm-system co-design. First, we replace dense agent interaction graphs with a hierarchical tree topology that induces structured sparsity in inter-agent communication. Second, we introduce a runtime adaptive mechanism that selectively terminates or skips downstream agent invocations using semantic agreement and confidence signals from intermediate outputs. Third, we pipeline agent execution by overlapping incremental prefilling with decoding across dependency-related agents, improving utilization and reducing inference latency. Across representative tasks, this approach substantially reduces end-to-end latency (up to 90%) while maintaining comparable accuracy (within $\pm$1%) relative to dense-connectivity MoA baselines, and can improve accuracy in certain settings.