MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMs

TL;DR

MoEEdit introduces a novel, routing-stable method for efficiently editing knowledge in sparse Mixture-of-Experts large language models, ensuring stability, specificity, and scalability.

Contribution

This work presents the first routing-stable knowledge editing framework for MoE LLMs, addressing stability and efficiency issues in existing methods.

Findings

Achieves state-of-the-art editing efficacy and generalization.

Maintains high routing stability and specificity.

Offers superior compute and memory efficiency.

Abstract

Knowledge editing (KE) enables precise modifications to factual content in large language models (LLMs). Existing KE methods are largely designed for dense architectures, limiting their applicability to the increasingly prevalent sparse Mixture-of-Experts (MoE) models that underpin modern scalable LLMs. Although MoEs offer strong efficiency and capacity scaling, naively adapting dense-model editors is both computationally costly and prone to routing distribution shifts that undermine stability and consistency. To address these challenges, we introduce MoEEdit, the first routing-stable framework for parameter-modifying knowledge editing in MoE LLMs. Our method reparameterizes expert updates via per-expert null-space projections that keep router inputs invariant and thereby suppress routing shifts. The resulting block-structured optimization is solved efficiently with a block coordinate descent (BCD) solver. Experiments show that MoEEdit attains state-of-the-art efficacy and generalization while preserving high specificity and routing stability, with superior compute and memory efficiency. These results establish a robust foundation for scalable, precise knowledge editing in sparse LLMs and underscore the importance of routing-stable interventions.