Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

TL;DR

The paper introduces Path-of-Thoughts (PoT), a framework that enhances large language models' relational reasoning by decomposing tasks into graph extraction, path identification, and reasoning, leading to improved accuracy and robustness.

Contribution

PoT is a novel approach that decomposes complex relational reasoning into graph-based steps, outperforming existing methods without fine-tuning or extensive LLM calls.

Findings

PoT surpasses state-of-the-art baselines by up to 21.3%

PoT is robust against extraction errors and input ambiguity

PoT does not require fine-tuning or extensive LLM calls

Abstract

Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework for solving relation reasoning that decomposes the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a reasoning graph that identifies crucial entities, relations, and attributes within the context. Subsequently, PoT identifies query-relevant reasoning paths within the graph, facilitating downstream reasoning of potential answers. Experimental evaluations across four datasets of relational reasoning demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (up to 21.3%) without requiring fine-tuning or extensive LLM calls. Furthermore, unlike prior neuro-symbolic methods, PoT exhibits improved resilience against LLM extraction errors and input ambiguity by leveraging the compositional nature of graphs.