Thought-Path Contrastive Learning via Premise-Oriented Data Augmentation for Logical Reading Comprehension

TL;DR

This paper introduces a premise-oriented data augmentation framework combined with thought-path contrastive learning to enhance logical reading comprehension in large language models, addressing previous limitations in reasoning and data diversity.

Contribution

It proposes a novel framework that generates rationales for both correct and incorrect options and constructs diverse counterfactual contexts, improving reasoning differentiation in LLMs.

Findings

Significant performance improvements on ReClor and LogiQA 2.0 benchmarks.

Effective differentiation of reasoning paths between original and counterfactual samples.

Enhanced reasoning capabilities in three representative large language models.

Abstract

Logical reading comprehension is a challenging task that entails grasping the underlying semantics of text and applying reasoning to deduce the correct answer. Prior researches have primarily focused on enhancing logical reasoning capabilities through Chain-of-Thought (CoT) or data augmentation. However, previous work constructing chain-of-thought rationales concentrates solely on analyzing correct options, neglecting the incorrect alternatives. Addtionally, earlier efforts on data augmentation by altering contexts rely on rule-based methods, which result in generated contexts that lack diversity and coherence. To address these issues, we propose a Premise-Oriented Data Augmentation (PODA) framework. This framework can generate CoT rationales including analyses for both correct and incorrect options, while constructing diverse and high-quality counterfactual contexts from incorrect candidate options. We integrate summarizing premises and identifying premises for each option into rationales. Subsequently, we employ multi-step prompts with identified premises to construct counterfactual context. To facilitate the model's capabilities to better differentiate the reasoning process associated with each option, we introduce a novel thought-path contrastive learning method that compares reasoning paths between the original and counterfactual samples. Experimental results on three representative LLMs demonstrate that our method can improve the baselines substantially across two challenging logical reasoning benchmarks (ReClor and LogiQA 2.0). The data and code are released at https://github.com/lalalamdbf/TPReasoner.