Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic

TL;DR

This paper introduces a formal logic-based synthetic corpus called FLD to evaluate and improve the deductive reasoning abilities of language models, revealing current limitations and guiding future research.

Contribution

It proposes a logically grounded deduction corpus based on formal logic, enabling more generalizable reasoning training and analysis for language models.

Findings

GPT-4 solves only half of the problems without additional training.

Training on FLD improves reasoning generalizability.

Certain reasoning aspects remain challenging for LLMs.

Abstract

We study a synthetic corpus based approach for language models (LMs) to acquire logical deductive reasoning ability. The previous studies generated deduction examples using specific sets of deduction rules. However, these rules were limited or otherwise arbitrary, limiting the generalizability of acquired reasoning ability. We rethink this and adopt a well-grounded set of deduction rules based on formal logic theory, which can derive any other deduction rules when combined in a multistep way. Then, using the proposed corpora, which we name FLD (Formal Logic Deduction), we first evaluate and analyze the logical reasoning ability of the latest LLMs. Even GPT-4 can solve only half of the problems, suggesting that pure logical reasoning isolated from knowledge is still challenging for the LLMs, and additional training specialized in logical reasoning is indeed essential. We next empirically verify that LMs trained on FLD corpora acquire more generalizable reasoning ability. Furthermore, we identify the aspects of reasoning ability on which deduction corpora can enhance LMs and those on which they cannot, and discuss future directions on each aspect. The released corpora serve both as learning resources and as challenging benchmarks.