Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debate

TL;DR

This paper introduces Self-Debate Reinforcement Learning (SDRL), a training framework that enhances large language models by enabling them to learn from self-debate, improving both standalone reasoning and multi-agent debate performance.

Contribution

SDRL is a novel training method that prepares models for multi-agent debate by jointly optimizing for standalone and debate-conditioned reasoning capabilities.

Findings

SDRL improves multi-agent debate performance across various protocols.

SDRL enhances single-model reasoning abilities.

Experiments show consistent gains across multiple models and benchmarks.

Abstract

The reasoning abilities of large language models (LLMs) have been substantially improved by reinforcement learning with verifiable rewards (RLVR). At test time, collaborative reasoning through Multi-Agent Debate (MAD) has emerged as a promising approach for enhancing LLM performance. However, current RLVR methods typically train LLMs to solve problems in isolation, without explicitly preparing them to synthesize and benefit from different rationales that arise during debate. In this work, we propose Self-Debate Reinforcement Learning(SDRL), a training framework where models learn from self-debate, equipping a single LLM with both strong standalone problem-solving ability and the capability to process diverse reasoning trajectories in MAD. Given a prompt, SDRL first samples multiple candidate solutions, then constructs a debate context with diverse reasoning paths and generates second-turn responses conditioned on this context. Finally, SDRL jointly optimizes both the initial and debate-conditioned responses, yielding a model that is effective as both a standalone solver and a debate participant. Experiments across multiple base models and reasoning benchmarks show that SDRL consistently improves MAD performance across diverse debate protocols and agent configurations, while simultaneously strengthening single-model reasoning.