Outdated Issue Aware Decoding for Reasoning Questions on Edited Knowledge

TL;DR

This paper introduces DISCO, a decoding strategy that improves reasoning in knowledge-edited models by emphasizing differences between original and edited models, significantly reducing outdated responses and enhancing reasoning accuracy.

Contribution

The paper proposes a novel decoding method, DISCO, that effectively mitigates outdated issue in knowledge editing by leveraging probability distribution differences.

Findings

DISCO outperforms previous methods by 12.99 F1 on reasoning questions.

Reduces outdated responses to 5.78% on the zsRE dataset.

Enhances reasoning capabilities of edited models.

Abstract

Recently, Knowledge Editing has received increasing attention, since it could update the specific knowledge from outdated ones in pretrained models without re-training. However, as pointed out by recent studies, existing related methods tend to merely memorize the superficial word composition of the edited knowledge, rather than truly learning and absorbing it. Consequently, on the reasoning questions, we discover that existing methods struggle to utilize the edited knowledge to reason the new answer, and tend to retain outdated responses, which are generated by the original models utilizing original knowledge. Nevertheless, the outdated responses are unexpected for the correct answers to reasoning questions, which we named as the outdated issue. To alleviate this issue, in this paper, we propose a simple yet effective decoding strategy, i.e., outDated ISsue aware deCOding (DISCO), to enhance the performance of edited models on reasoning questions. Specifically, we capture the difference in the probability distribution between the original and edited models. Further, we amplify the difference of the token prediction in the edited model to alleviate the outdated issue, and thus enhance the model performance w.r.t the edited knowledge. Experimental results suggest that applying DISCO could enhance edited models to reason, e.g., on reasoning questions, DISCO outperforms the prior SOTA method by 12.99 F1 scores, and reduces the ratio of the outdated issue to 5.78% on the zsRE dataset.