Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate

TL;DR

This paper introduces SIAR and R3, two methods leveraging large language models to enhance rule retrieval and relevance estimation, significantly improving reasoning accuracy by addressing semantic gaps and better aligning rules with query facts.

Contribution

The paper proposes Self-Induction Augmented Retrieval (SIAR) and Rule Relevance ReEstimate (R3), novel techniques that improve rule retrieval and relevance assessment for reasoning tasks using LLMs.

Findings

SIAR improves rule retrieval accuracy in reasoning tasks.

R3 enhances the relevance estimation of retrieved rules.

Experiments show significant performance gains across various settings.

Abstract

This paper systematically addresses the challenges of rule retrieval, a crucial yet underexplored area. Vanilla retrieval methods using sparse or dense retrievers to directly search for relevant rules to support downstream reasoning, often suffer from low accuracy. This is primarily due to a significant semantic gap between the instantiated facts in the queries and the abstract representations of the rules. Such misalignment results in suboptimal retrieval quality, which in turn negatively impacts reasoning performance. To overcome these challenges, we propose Self-Induction Augmented Retrieval (SIAR), a novel approach that utilizes Large Language Models (LLMs) to induce potential inferential rules that might offer benefits for reasoning by abstracting the underlying knowledge and logical structure in queries. These induced rules are then used for query augmentation to improve retrieval effectiveness. Additionally, we introduce Rule Relevance ReEstimate (R$^3$), a method that re-estimates the relevance of retrieved rules by assessing whether the abstract knowledge they contain can be instantiated to align with the facts in the queries and the helpfulness for reasoning. Extensive experiments across various settings demonstrate the effectiveness and versatility of our proposed methods.