Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

TL;DR

This paper introduces Logic-of-Thought prompting, a method that uses propositional logic to expand logical information in contexts, significantly improving the reasoning performance of large language models across multiple tasks.

Contribution

The paper proposes a novel Logic-of-Thought prompting technique that enhances logical reasoning in LLMs by ensuring information completeness and can be combined with existing prompting methods.

Findings

LoT improves reasoning accuracy across five tasks.

Significant performance gains on ReClor, RuleTaker, and ProofWriter datasets.

LoT boosts Chain-of-Thought and Tree-of-Thoughts methods.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Although some prompting methods, such as Chain-of-Thought, can improve the reasoning ability of LLMs to some extent, they suffer from an unfaithful issue where derived conclusions may not align with the generated reasoning chain. To address this issue, some studies employ the approach of propositional logic to further enhance logical reasoning abilities of LLMs. However, the potential omissions in the extraction of logical expressions in these methods can cause information loss in the logical reasoning process, thereby generating incorrect results. To this end, we propose Logic-of-Thought (LoT) prompting which employs propositional logic to generate expanded logical information descriptions and utilizes them as an additional augmentation to original contexts, thereby ensuring information completeness and enhancing logical reasoning ability. LoT is orthogonal to existing prompting methods and can be seamlessly integrated with them. Extensive experiments demonstrate that LoT boosts the performance of various prompting methods with a striking margin across five logical reasoning tasks. In particular, LoT enhances Chain-of-Thought's performance on the ReClor dataset by +4.35%, improves Chain-of-Thought with Self-Consistency's performance on the RuleTaker dataset by +3.52%, and boosts performance of Tree-of-Thoughts on the ProofWriter dataset by +8%.