MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors

TL;DR

MEMoE introduces a mixture of experts approach with knowledge routing to improve model editing in large language models, balancing generalization and locality without altering original parameters.

Contribution

The paper presents MEMoE, a novel model editing adapter using MoE architecture and knowledge anchor routing to enhance editing performance and preserve model capabilities.

Findings

Outperforms existing editing methods in various tasks.

Balances generalization and locality effectively.

Maintains original model parameters while updating knowledge.

Abstract

Model editing aims to efficiently alter the behavior of Large Language Models (LLMs) within a desired scope, while ensuring no adverse impact on other inputs. Recent years have witnessed various model editing methods been proposed. However, these methods either exhibit poor overall performance or struggle to strike a balance between generalization and locality. We propose MEMoE, a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. MEMoE updates knowledge using a bypass MoE structure, keeping the original parameters unchanged to preserve the general ability of LLMs. And, the knowledge anchor routing ensures that inputs requiring similar knowledge are routed to the same expert, thereby enhancing the generalization of the updated knowledge. Experimental results show the superiority of our approach over both batch editing and sequential batch editing tasks, exhibiting exceptional overall performance alongside outstanding balance between generalization and locality. Our code will be available.