Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learning

TL;DR

This paper introduces ODEdit, a novel invariant learning framework that improves knowledge editing in Multimodal Large Language Models by focusing on invariant causal trajectories, leading to more reliable and generalizable edits.

Contribution

It reformulates MLLM editing as an out-of-distribution generalization problem and proposes a new invariant learning method with theoretical analysis and extensive experiments.

Findings

ODEdit enhances editing reliability, locality, and generality.

The invariant trajectory learning method stabilizes edits against environmental variations.

Theoretical analysis confirms the effectiveness of the proposed approach.

Abstract

Knowledge editing emerges as a crucial technique for efficiently correcting incorrect or outdated knowledge in large language models (LLM). Existing editing methods rely on a rigid mapping from parameter or module modifications to output, which causes the generalization limitation in Multimodal LLM (MLLM). In this paper, we reformulate MLLM editing as an out-of-distribution (OOD) generalization problem, where the goal is to discern semantic shift with factual shift and thus achieve robust editing among diverse cross-modal prompting. The key challenge of this OOD problem lies in identifying invariant causal trajectories that generalize accurately while suppressing spurious correlations. To address it, we propose ODEdit, a plug-and-play invariant learning based framework that optimizes the tripartite OOD risk objective to simultaneously enhance editing reliability, locality, and generality.We further introduce an edit trajectory invariant learning method, which integrates a total variation penalty into the risk minimization objective to stabilize edit trajectories against environmental variations. Theoretical analysis and extensive experiments demonstrate the effectiveness of ODEdit.