IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning

TL;DR

IDOL introduces a simple yet effective pre-training method using logical indicators and a rich dataset to significantly improve logical reasoning in machine reading comprehension models, outperforming existing benchmarks and large language models.

Contribution

The paper proposes IDOL, a novel pre-training task that enhances logical reasoning in models through indicator-based training and a specialized dataset, achieving state-of-the-art results.

Findings

IDOL outperforms existing models on ReClor and LogiQA benchmarks.

IDOL generalizes well across different pre-trained models and MRC tasks.

IDOL maintains competitive performance on general language understanding tasks.

Abstract

In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LGP (LoGic Pre-training). IDOL achieves state-of-the-art performance on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage.