Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

TL;DR

This paper introduces a scalable, tensor-structured knowledge editing method for Mixture-of-Experts LLMs that is efficient, accurate, and extends KE capabilities to sparse architectures.

Contribution

It develops a novel tensor-based framework for KE in MoE LLMs, leveraging the Woodbury identity for efficient low-rank updates, matching quality of strong baselines while being significantly faster.

Findings

Matches baseline KE quality on main metrics

Accelerates editing by up to 6x

Extends KE to sparse MoE architectures

Abstract

Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE) architectures for their superior memory footprint and inference efficiency. This mismatch leaves a growing class of production models without principled editing tools. We propose a MEMIT-like framework for knowledge editing in MoE-based LLMs. Our method exploits the tensor structure of MoE layers to formulate the editing objective faithfully at the per expert level, and applies the Woodbury matrix identity to avoid materializing or inverting the full stacked matrix of expert weights. The resulting update reduces to inversions of fixed low-rank matrices and requires no additional backward passes. Empirically, our approach matches the editing quality of strong baselines on the main KE metrics while accelerating the editing procedure by up to 6x, owing to the batched MEMIT-style formulation and the low-dimensional inversions enabled by the Woodbury identity. These results show that closed-form, parameter-modifying KE can be extended efficiently beyond dense layers, opening a path toward scalable knowledge editing in modern sparse LLM architectures.