ReasonEdit: Editing Vision-Language Models using Human Reasoning

TL;DR

ReasonEdit is a novel vision-language model editor that incorporates human reasoning explanations, improving editing accuracy and generalization on reasoning-heavy visual question answering tasks.

Contribution

It introduces a new editing setup that stores human reasoning in a codebook and uses a topology-balanced embedding, achieving state-of-the-art results.

Findings

ReasonEdit outperforms existing editors on multiple datasets.

Using human reasoning during editing enhances generalization.

The approach effectively handles reasoning-heavy vision-language tasks.

Abstract

Model editing aims to correct errors in large, pretrained models without altering unrelated behaviors. While some recent works have edited vision-language models (VLMs), no existing editors tackle reasoning-heavy tasks, which typically require humans and models to reason about images. We therefore propose ReasonEdit, the first VLM editor to let users explain their reasoning during editing, introducing a new, practical model editing setup. ReasonEdit continuously stores human reasoning in a codebook, and retrieves only relevant facts during inference using a novel topology-balanced multimodal embedding method inspired by network science. Across four VLMs on multiple rationale-based visual question answering datasets, ReasonEdit achieves state-of-the-art editing performance, ultimately showing that using human reasoning during editing greatly improves edit generalization.