LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs

TL;DR

This paper introduces EBench-18K, a large-scale benchmark for Text-guided Image Editing evaluation, and proposes LMM4Edit, a multimodal model-based metric that aligns well with human preferences across multiple editing quality aspects.

Contribution

The paper presents the first large-scale benchmark for TIE evaluation and introduces LMM4Edit, a novel multimodal metric that comprehensively assesses image editing quality and alignment.

Findings

LMM4Edit outperforms existing metrics in correlating with human preferences.

EBench-18K provides extensive data for evaluating TIE models across multiple dimensions.

LMM4Edit demonstrates strong generalization in zero-shot evaluations.

Abstract

The rapid advancement of Text-guided Image Editing (TIE) enables image modifications through text prompts. However, current TIE models still struggle to balance image quality, editing alignment, and consistency with the original image, limiting their practical applications. Existing TIE evaluation benchmarks and metrics have limitations on scale or alignment with human perception. To this end, we introduce EBench-18K, the first large-scale image Editing Benchmark including 18K edited images with fine-grained human preference annotations for evaluating TIE. Specifically, EBench-18K includes 1,080 source images with corresponding editing prompts across 21 tasks, 18K+ edited images produced by 17 state-of-the-art TIE models, 55K+ mean opinion scores (MOSs) assessed from three evaluation dimensions, and 18K+ question-answering (QA) pairs. Based on EBench-18K, we employ outstanding LMMs to assess edited images, while the evaluation results, in turn, provide insights into assessing the alignment between the LMMs' understanding ability and human preferences. Then, we propose LMM4Edit, a LMM-based metric for evaluating image Editing models from perceptual quality, editing alignment, attribute preservation, and task-specific QA accuracy in an all-in-one manner. Extensive experiments show that LMM4Edit achieves outstanding performance and aligns well with human preference. Zero-shot validation on the other datasets also shows the generalization ability of our model. The dataset and code are available at https://github.com/IntMeGroup/LMM4Edit.