DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models

TL;DR

DualEdit is a novel method for updating knowledge in vision-language models by editing both textual and visual modalities at their respective sensitive layers, improving knowledge update efficiency while preserving original capabilities.

Contribution

The paper introduces DualEdit, the first editing approach that modifies both modalities in VLMs at their key layers, with a gating module to balance knowledge updating and retention.

Findings

DualEdit outperforms state-of-the-art VLM editing methods.

Editing both modalities enhances knowledge update efficiency.

The gating module helps preserve original model capabilities.

Abstract

Model editing aims to efficiently update a pre-trained model's knowledge without the need for time-consuming full retraining. While existing pioneering editing methods achieve promising results, they primarily focus on editing single-modal language models (LLMs). However, for vision-language models (VLMs), which involve multiple modalities, the role and impact of each modality on editing performance remain largely unexplored. To address this gap, we explore the impact of textual and visual modalities on model editing and find that: (1) textual and visual representations reach peak sensitivity at different layers, reflecting their varying importance; and (2) editing both modalities can efficiently update knowledge, but this comes at the cost of compromising the model's original capabilities. Based on our findings, we propose DualEdit, an editor that modifies both textual and visual modalities at their respective key layers. Additionally, we introduce a gating module within the more sensitive textual modality, allowing DualEdit to efficiently update new knowledge while preserving the model's original information. We evaluate DualEdit across multiple VLM backbones and benchmark datasets, demonstrating its superiority over state-of-the-art VLM editing baselines as well as adapted LLM editing methods on different evaluation metrics. Codes are available at https://github.com/zhiyiscs/DualEdit