Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models

TL;DR

This paper introduces a relation-focused approach to improve knowledge editing in transformer language models, addressing over-generalization issues by considering relation information during recall.

Contribution

It proposes a novel relation-centric interpretation of knowledge recall and demonstrates its effectiveness in reducing over-generalization in knowledge editing.

Findings

Significantly reduces over-generalization in knowledge editing.

Maintains competitive performance on other evaluation criteria.

Introduces a new R-Specificity criterion for assessment.

Abstract

The storage and recall of factual associations in auto-regressive transformer language models (LMs) have drawn a great deal of attention, inspiring knowledge editing by directly modifying the located model weights. Most editing works achieve knowledge editing under the guidance of existing interpretations of knowledge recall that mainly focus on subject knowledge. However, these interpretations are seriously flawed, neglecting relation information and leading to the over-generalizing problem for editing. In this work, we discover a novel relation-focused perspective to interpret the knowledge recall of transformer LMs during inference and apply it on single knowledge editing to avoid over-generalizing. Experimental results on the dataset supplemented with a new R-Specificity criterion demonstrate that our editing approach significantly alleviates over-generalizing while remaining competitive on other criteria, breaking the domination of subject-focused editing for future research.