Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse

TL;DR

This paper analyzes how sequential knowledge editing affects large language models, revealing spectral sensitivities and proposing REVIVE to preserve model abilities during extensive editing.

Contribution

It introduces a spectral analysis framework linking model abilities to singular directions and proposes REVIVE to stabilize sequential knowledge editing.

Findings

REVIVE improves editing efficacy across models and benchmarks.

Spectral analysis reveals dominant singular directions are sensitive to edits.

REVIVE maintains model general abilities even after 20,000 edits.

Abstract

Sequential knowledge editing in large language models often causes catastrophic collapse of the model's general abilities, especially for parameter-modifying methods. Existing approaches mitigate this issue through heuristic constraints on parameter updates, yet the mechanisms underlying such degradation remain insufficiently understood. In this work, we present a spectral analysis of sequential knowledge editing and show that a model's general abilities are closely associated with dominant singular directions of pretrained weight matrices. These directions are highly sensitive to perturbations and are progressively disrupted by repeated edits, closely tracking the collapse in both editing efficacy and general performance. Building on this insight, we propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE represents parameter updates in the spectral basis of the original weights and filters components that would interfere with the protected region. Extensive experiments across multiple models and benchmarks show that REVIVE consistently improves editing efficacy while substantially preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits.