Norm Growth and Stability Challenges in Localized Sequential Knowledge Editing

TL;DR

This paper explores how localized knowledge editing in large language models causes growth in matrix norms, leading to stability issues and altered internal representations, which challenge model maintenance and performance.

Contribution

It uncovers the consistent increase in Frobenius norms during various editing techniques and analyzes the resulting stability and representational shifts in LLMs.

Findings

Matrix norms increase with successive updates across editing methods.

Activation vectors shift to different subspaces, indicating representational changes.

Norm growth correlates with potential instability and performance degradation.

Abstract

This study investigates the impact of localized updates to large language models (LLMs), specifically in the context of knowledge editing - a task aimed at incorporating or modifying specific facts without altering broader model capabilities. We first show that across different post-training interventions like continuous pre-training, full fine-tuning and LORA-based fine-tuning, the Frobenius norm of the updated matrices always increases. This increasing norm is especially detrimental for localized knowledge editing, where only a subset of matrices are updated in a model . We reveal a consistent phenomenon across various editing techniques, including fine-tuning, hypernetwork-based approaches, and locate-and-edit methods: the norm of the updated matrix invariably increases with successive updates. Such growth disrupts model balance, particularly when isolated matrices are updated while the rest of the model remains static, leading to potential instability and degradation of downstream performance. Upon deeper investigations of the intermediate activation vectors, we find that the norm of internal activations decreases and is accompanied by shifts in the subspaces occupied by these activations, which shows that these activation vectors now occupy completely different regions in the representation space compared to the unedited model. With our paper, we highlight the technical challenges with continuous and localized sequential knowledge editing and their implications for maintaining model stability and utility.