RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation

TL;DR

RMoA introduces a novel multi-agent framework with residual connections and diversity mechanisms to enhance efficiency, robustness, and performance across various tasks, while reducing computational costs.

Contribution

It presents a residual-based multi-agent system with diversity selection and adaptive termination, achieving state-of-the-art results with lower computational overhead.

Findings

State-of-the-art performance on multiple benchmarks.

Significant reduction in computational overhead.

Effective information preservation through residual mechanisms.

Abstract

Although multi-agent systems based on large language models show strong capabilities on multiple tasks, they are still limited by high computational overhead, information loss, and robustness. Inspired by ResNet's residual learning, we propose Residual Mixture-of-Agents (RMoA), integrating residual connections to optimize efficiency and reliability. To maximize information utilization from model responses while minimizing computational costs, we innovatively design an embedding-based diversity selection mechanism that greedily selects responses via vector similarity. Furthermore, to mitigate iterative information degradation, we introduce a Residual Extraction Agent to preserve cross-layer incremental information by capturing inter-layer response differences, coupled with a Residual Aggregation Agent for hierarchical information integration. Additionally, we propose an adaptive termination mechanism that dynamically halts processing based on residual convergence, further improving inference efficiency. RMoA achieves state-of-the-art performance on the benchmarks of across alignment, mathematical reasoning, code generation, and multitasking understanding, while significantly reducing computational overhead. Code is available at https://github.com/mindhunter01/RMoA.