LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning

TL;DR

The paper introduces LINKED, a novel method that enhances large language models' commonsense reasoning by filtering noisy knowledge and reducing invalid reasoning, leading to significant accuracy improvements on benchmark tasks.

Contribution

The paper presents a new approach combining knowledge filtering and reasoning modules, along with a novel effectiveness-preservation score to evaluate knowledge injection impacts.

Findings

Outperforms state-of-the-art methods by up to 9.0% accuracy

Effectiveness-preservation score effectively measures knowledge impact

Extensive analysis reveals insights into LLMs' reasoning capabilities

Abstract

Large language models (LLMs) sometimes demonstrate poor performance on knowledge-intensive tasks, commonsense reasoning is one of them. Researchers typically address these issues by retrieving related knowledge from knowledge graphs or employing self-enhancement methods to elicit knowledge in LLMs. However, noisy knowledge and invalid reasoning issues hamper their ability to answer questions accurately. To this end, we propose a novel method named eliciting, filtering and integrating knowledge in large language model (LINKED). In it, we design a reward model to filter out the noisy knowledge and take the marginal consistent reasoning module to reduce invalid reasoning. With our comprehensive experiments on two complex commonsense reasoning benchmarks, our method outperforms SOTA baselines (up to 9.0% improvement of accuracy). Besides, to measure the positive and negative impact of the injected knowledge, we propose a new metric called effectiveness-preservation score for the knowledge enhancement works. Finally, through extensive experiments, we conduct an in-depth analysis and find many meaningful conclusions about LLMs in commonsense reasoning tasks.