Adaptive Robust Estimator for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a robust multi-agent reinforcement learning framework with a structured reasoning pipeline and an adaptive estimator, improving stability and performance in noisy reward environments.

Contribution

It proposes the DACR framework and ARE estimator, enhancing credit attribution and robustness in multi-agent collaborative reasoning tasks.

Findings

Outperforms baselines on mathematical reasoning benchmarks

Demonstrates robustness to reward noise in experiments

Achieves more stable training dynamics

Abstract

Multi-agent collaboration has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models, yet it suffers from interaction-level ambiguity that blurs generation, critique, and revision, making credit assignment across agents difficult. Moreover, policy optimization in this setting is vulnerable to heavy-tailed and noisy rewards, which can bias advantage estimation and trigger unstable or even divergent training. To address both issues, we propose a robust multi-agent reinforcement learning framework for collaborative reasoning, consisting of two components: Dual-Agent Answer-Critique-Rewrite (DACR) and an Adaptive Robust Estimator (ARE). DACR decomposes reasoning into a structured three-stage pipeline: answer, critique, and rewrite, while enabling explicit attribution of each agent's marginal contribution to its partner's performance. ARE provides robust estimation of batch experience means during multi-agent policy optimization. Across mathematical reasoning and embodied intelligence benchmarks, even under noisy rewards, our method consistently outperforms the baseline in both homogeneous and heterogeneous settings. These results indicate stronger robustness to reward noise and more stable training dynamics, effectively preventing optimization failures caused by noisy reward signals.