Causal Reasoning in Large Language Models: A Knowledge Graph Approach

TL;DR

This paper introduces a knowledge graph-based reasoning method that leverages causal relationships to enhance large language models' reasoning and performance, demonstrating the importance of causal structures in prompt design.

Contribution

It proposes a novel KG-based random-walk reasoning approach that effectively incorporates causal relationships, improving LLM reasoning and performance on commonsense question answering tasks.

Findings

KG-based reasoning improves LLM performance

Adding irrelevant sentences via KG enhances reasoning

Causal structures significantly boost reasoning capabilities

Abstract

Large language models (LLMs) typically improve performance by either retrieving semantically similar information, or enhancing reasoning abilities through structured prompts like chain-of-thought. While both strategies are considered crucial, it remains unclear which has a greater impact on model performance or whether a combination of both is necessary. This paper answers this question by proposing a knowledge graph (KG)-based random-walk reasoning approach that leverages causal relationships. We conduct experiments on the commonsense question answering task that is based on a KG. The KG inherently provides both relevant information, such as related entity keywords, and a reasoning structure through the connections between nodes. Experimental results show that the proposed KG-based random-walk reasoning method improves the reasoning ability and performance of LLMs. Interestingly, incorporating three seemingly irrelevant sentences into the query using KG-based random-walk reasoning enhances LLM performance, contrary to conventional wisdom. These findings suggest that integrating causal structures into prompts can significantly improve reasoning capabilities, providing new insights into the role of causality in optimizing LLM performance.