Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planning

TL;DR

This paper introduces CodePlan, a scalable framework enabling large language models to generate structured code-form plans for improved multi-step reasoning, achieving significant performance gains across diverse reasoning benchmarks.

Contribution

We propose CodePlan, a novel method that leverages code-form plans for reasoning, trained on a large-scale dataset, enhancing LLM reasoning capabilities without task-specific fine-tuning.

Findings

25.1% relative improvement over baseline models

Effective across diverse reasoning tasks

Significant gains on complex reasoning benchmarks

Abstract

Despite the remarkable success of large language models (LLMs) on traditional natural language processing tasks, their planning ability remains a critical bottleneck in tackling complex multi-step reasoning tasks. Existing approaches mainly rely on prompting or task-specific fine-tuning, often suffering from poor robustness and cross-task generalization. To address the limitation, we introduce CodePlan, a scalable framework that empowers LLMs to generate and follow \textit{code-form plans} -- pseudocode that outlines high-level, structured reasoning processes. By leveraging the structured and versatile nature of code, CodePlan effectively captures the rich semantics and control flows inherent to sophisticated reasoning tasks. Importantly, CodePlan allows automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. This enables it to scale up efficiently and improve LLM's reasoning capabilities across diverse scenarios. To train CodePlan, we construct a large-scale dataset of 2M examples that integrate code-form plans with standard prompt-response pairs from existing corpora. With minimal computation overhead during both training and inference, CodePlan achieves a 25.1\% relative improvement compared with directly generating responses, averaged across 13 challenging multi-step reasoning benchmarks, spanning mathematical reasoning, symbolic reasoning, instruction-following, multi-hop QA, and decision-making tasks. Further analysis reveals CodePlan's increasing performance gains on more complex reasoning tasks, as well as significant data efficiency thanks to its generalization ability.