Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

TL;DR

This paper introduces MATTRL, a test-time reinforcement learning framework for multi-agent systems that enhances reasoning accuracy across various domains by integrating structured experiences during inference, improving robustness and efficiency.

Contribution

The paper proposes a novel test-time reinforcement learning approach for multi-agent systems, enabling stable, distribution-shift-robust reasoning without additional tuning.

Findings

MATTRL improves accuracy by 3.67% over multi-agent baselines.

MATTRL improves accuracy by 8.67% over single-agent baselines.

Ablation studies reveal the impact of different credit-assignment schemes.

Abstract

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.