MetaKE: Meta-Learning for Knowledge Editing Toward a Better Accuracy-Editability Trade-off

TL;DR

MetaKE introduces a bi-level meta-learning framework for knowledge editing that unifies upstream and downstream optimization, improving the accuracy-editability trade-off in language models.

Contribution

It proposes a novel meta-learning approach with a structural gradient proxy to better balance accuracy and editability in knowledge editing tasks.

Findings

MetaKE outperforms existing KE methods in experiments.

The framework effectively balances semantic accuracy and editability.

The structural gradient proxy reduces computational costs.

Abstract

Existing locate-then-edit Knowledge Editing (KE) methods typically decompose editing into two stages: upstream target representation optimization and downstream constrained parameter optimization. The optimization across the two stages is disconnected: upstream applies uniform regularization without observing downstream realization of the planned residual, hindering a refined accuracy-editability trade-off. Since this realization is request-specific and depends on downstream constraints, uniform regularization can over-shrink high-association requests, causing insufficient editing, while it can under-regularize low-association requests, producing over-large planned residuals that reduce downstream editability. To bridge this disconnect, we propose MetaKE (Meta-learning for Knowledge Editing), a new framework that unifies upstream and downstream stages into a bi-level optimization problem. The inner level optimizes parameter updates for the target representation, while the outer level optimizes representation using feedback from downstream constraints, achieving a better semantic accuracy-editability trade-off. To avoid costly multi-layer backpropagation, we introduce a Structural Gradient Proxy to approximate and propagate this feedback. Extensive experiments show that MetaKE outperforms strong baselines, offering a new perspective on KE.