Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysis

TL;DR

This paper introduces CHECK, a semantic analysis-based knowledge editing framework for multi-hop question answering, which improves reasoning accuracy by analyzing and revising reasoning chains before executing answers.

Contribution

CHECK leverages semantic analysis and logic optimization to enhance multi-hop reasoning in LLMs, addressing limitations of existing knowledge editors.

Findings

Achieves 22.8% average improvement in MQA accuracy

Effective in revising reasoning chains for logical consistency

Outperforms five state-of-the-art frameworks on four datasets

Abstract

Large Language Models (LLMs) require lightweight avenues of updating stored information that has fallen out of date. Knowledge Editing (KE) approaches have been successful in updating model knowledge for simple factual queries but struggle with handling tasks that require compositional reasoning such as multi-hop question answering (MQA). We observe that existing knowledge editors leverage decompositional techniques that result in illogical reasoning processes. In this paper, we propose a knowledge editor for MQA based on semantic analysis called CHECK. Our framework is based on insights from an analogy between compilers and reasoning using LLMs. Similar to how source code is first compiled before being executed, we propose to semantically analyze reasoning chains before executing the chains to answer questions. Reasoning chains with semantic errors are revised to ensure consistency through logic optimization and re-prompting the LLM model at a higher temperature. We evaluate the effectiveness of CHECK against five state-of-the-art frameworks on four datasets and achieve an average 22.8% improved MQA accuracy.