Cooperative Strategic Planning Enhances Reasoning Capabilities in Large Language Models

TL;DR

This paper introduces CoPlanner, a cooperative multi-agent framework with distinct planning and reasoning agents, significantly improving reasoning performance in large language models on complex tasks.

Contribution

The paper presents a novel multi-agent reasoning framework that separates planning and reasoning, training the planning agent via reinforcement learning for enhanced cooperation.

Findings

Outperforms previous methods by 9.94% on LogiQA

Achieves 3.09% improvement on BBH

Demonstrates effective cooperation boosts multi-step reasoning

Abstract

Enhancing the reasoning capabilities of large language models (LLMs) is crucial for enabling them to tackle complex, multi-step problems. Multi-agent frameworks have shown great potential in enhancing LLMs' reasoning capabilities. However, the lack of effective cooperation between LLM agents hinders their performance, especially for multi-step reasoning tasks. This paper proposes a novel cooperative multi-agent reasoning framework (CoPlanner) by separating reasoning steps and assigning distinct duties to different agents. CoPlanner consists of two LLM agents: a planning agent and a reasoning agent. The planning agent provides high-level strategic hints, while the reasoning agent follows these hints and infers answers. By training the planning agent's policy through the interactive reasoning process via Proximal Policy Optimization (PPO), the LLaMA-3-8B-based CoPlanner outperforms the previous best method by 9.94\% on LogiQA and 3.09\% on BBH. Our results demonstrate that the guidance from the planning agent and the effective cooperation between the agents contribute to the superior performance of CoPlanner in tackling multi-step reasoning problems.