Reversible Lifelong Model Editing via Semantic Routing-Based LoRA

TL;DR

This paper introduces SoLA, a novel framework for lifelong model editing in large language models that enables precise, reversible, and efficient updates through semantic routing and independent LoRA modules.

Contribution

The paper presents a reversible lifelong model editing method using semantic routing-based LoRA, addressing semantic drift and knowledge forgetting, with the ability to revoke specific edits.

Findings

Effective learning and retention of edited knowledge.

Achieves accurate, efficient, and reversible model updates.

First to enable reversible rollback in lifelong model editing.

Abstract

The dynamic evolution of real-world necessitates model editing within Large Language Models. While existing methods explore modular isolation or parameter-efficient strategies, they still suffer from semantic drift or knowledge forgetting due to continual updating. To address these challenges, we propose SoLA, a Semantic routing-based LoRA framework for lifelong model editing. In SoLA, each edit is encapsulated as an independent LoRA module, which is frozen after training and mapped to input by semantic routing, allowing dynamic activation of LoRA modules via semantic matching. This mechanism avoids semantic drift caused by cluster updating and mitigates catastrophic forgetting from parameter sharing. More importantly, SoLA supports precise revocation of specific edits by removing key from semantic routing, which restores model's original behavior. To our knowledge, this reversible rollback editing capability is the first to be achieved in existing literature. Furthermore, SoLA integrates decision-making process into edited layer, eliminating the need for auxiliary routing networks and enabling end-to-end decision-making process. Extensive experiments demonstrate that SoLA effectively learns and retains edited knowledge, achieving accurate, efficient, and reversible lifelong model editing.