What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models

TL;DR

This paper introduces DICE, a multimodal model that detects and evaluates localized image edits based on instructions, improving alignment with human judgment and explainability in image editing evaluation.

Contribution

The paper presents DICE, a novel model combining difference detection and coherence estimation for instruction-guided image edits, trained with self-supervision and distillation techniques.

Findings

DICE accurately detects relevant image differences.

DICE's evaluations strongly correlate with human judgments.

The framework outperforms existing metrics in assessing image edits.

Abstract

Instruction-based image editing models offer increased personalization opportunities in generative tasks. However, properly evaluating their results is challenging, and most of the existing metrics lag in terms of alignment with human judgment and explainability. To tackle these issues, we introduce DICE (DIfference Coherence Estimator), a model designed to detect localized differences between the original and the edited image and to assess their relevance to the given modification request. DICE consists of two key components: a difference detector and a coherence estimator, both built on an autoregressive Multimodal Large Language Model (MLLM) and trained using a strategy that leverages self-supervision, distillation from inpainting networks, and full supervision. Through extensive experiments, we evaluate each stage of our pipeline, comparing different MLLMs within the proposed framework. We demonstrate that DICE effectively identifies coherent edits, effectively evaluating images generated by different editing models with a strong correlation with human judgment. We publicly release our source code, models, and data.