Explicit Planning Helps Language Models in Logical Reasoning

TL;DR

LEAP introduces explicit planning into language models for multi-step logical reasoning, significantly improving performance on standard datasets and outperforming larger models like GPT-3 in certain tasks.

Contribution

The paper presents LEAP, a novel system integrating explicit planning into language models for logical reasoning, with a new training strategy to prevent spurious features from misleading the process.

Findings

LEAP outperforms competing methods on multiple datasets.

Small T5 models with LEAP perform competitively with GPT-3.

LEAP surpasses chain-of-thought prompting on PrOntoQA.

Abstract

Language models have been shown to perform remarkably well on a wide range of natural language processing tasks. In this paper, we propose LEAP, a novel system that uses language models to perform multi-step logical reasoning and incorporates explicit planning into the inference procedure. Explicit planning enables the system to make more informed reasoning decisions at each step by looking ahead into their future effects. Moreover, we propose a training strategy that safeguards the planning process from being led astray by spurious features. Our full system significantly outperforms other competing methods on multiple standard datasets. When using small T5 models as its core selection and deduction components, our system performs competitively compared to GPT-3 despite having only about 1B parameters (i.e., 175 times smaller than GPT-3). When using GPT-3.5, it significantly outperforms chain-of-thought prompting on the challenging PrOntoQA dataset. We have conducted extensive empirical studies to demonstrate that explicit planning plays a crucial role in the system's performance.