DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs

TL;DR

DRAFT-RL introduces a multi-agent reinforcement learning framework with Chain-of-Draft reasoning, enabling diverse multi-path exploration and peer-guided reflection to improve complex reasoning tasks in LLMs.

Contribution

It integrates Chain-of-Draft reasoning into multi-agent RL, allowing multiple drafts and peer evaluation for enhanced reasoning robustness and interpretability.

Findings

Outperforms existing reflective agents in accuracy

Achieves faster convergence in complex tasks

Enhances reasoning robustness through multi-path exploration

Abstract

Large Language Models (LLMs) have shown impressive capabilities in multi-step reasoning and problem-solving.Recent works introduce multi-agent reflection frameworks where multiple LLM agents critique and refine each other's outputs using reinforcement learning (RL). However, these approaches often rely on single-shot responses and lack structural diversity in reasoning exploration. In this paper, we propose DRAFT-RL, a novel framework that integrates Chain-of-Draft (CoD) reasoning into multi-agent RL training. Instead of generating single responses, each agent produces multiple drafts per query, which are then evaluated by peer agents and a learned reward model to identify the most promising trajectory. These selected drafts are used to refine future reasoning strategies through actor-critic learning.DRAFT-RL enables explicit multi-path exploration, peer-guided reflection, and reward-aligned selection, resulting in more robust and interpretable LLM agent behavior. We evaluate our method on complex reasoning tasks including code synthesis, symbolic math, and knowledge-intensive QA,demonstrating that DRAFT-RL outperforms existing reflective and RL-based agents by significant margins in both accuracy and convergence speed