On the Limitations of Rank-One Model Editing in Answering Multi-hop Questions

TL;DR

This paper investigates the limitations of Rank-One Model Editing (ROME) in multi-hop question answering, identifying failure modes and proposing Redundant Editing to significantly improve multi-hop reasoning accuracy.

Contribution

The paper reveals key failure modes of ROME in multi-hop reasoning and introduces Redundant Editing, a strategy that substantially enhances multi-hop question answering performance.

Findings

Redundant Editing improves 2-hop question accuracy by at least 15.5 percentage points.

It achieves a 96% increase over previous single-edit strategies.

The method trades off some specificity and naturalness for better reasoning accuracy.

Abstract

Recent advances in Knowledge Editing (KE), particularly Rank-One Model Editing (ROME), show superior efficiency over fine-tuning and in-context learning for updating single-hop facts in transformers. However, these methods face significant challenges when applied to multi-hop reasoning tasks requiring knowledge chaining. In this work, we study the effect of editing knowledge with ROME on different layer depths and identify three key failure modes. First, the "hopping-too-late" problem occurs as later layers lack access to necessary intermediate representations. Second, generalization ability deteriorates sharply when editing later layers. Third, the model overfits to edited knowledge, incorrectly prioritizing edited-hop answers regardless of context. To mitigate the issues of "hopping-too-late" and generalisation decay, we propose Redundant Editing, a simple yet effective strategy that enhances multi-hop reasoning. Our experiments demonstrate that this approach can improve accuracy on 2-hop questions by at least 15.5 percentage points, representing a 96% increase over the previous single-edit strategy, while trading off some specificity and language naturalness.