I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Models

TL;DR

I2I-Bench is a comprehensive, multi-task benchmark suite for evaluating image-to-image editing models across diverse tasks and evaluation dimensions, combining automated tools and human validation.

Contribution

The paper introduces I2I-Bench, a new benchmark with diverse tasks, detailed evaluation metrics, and automated assessment methods for image editing models.

Findings

Benchmark reveals gaps and trade-offs among existing models.

Automated evaluation methods align well with human preferences.

I2I-Bench covers 10 task categories and 30 evaluation dimensions.

Abstract

Image editing models are advancing rapidly, yet comprehensive evaluation remains a significant challenge. Existing image editing benchmarks generally suffer from limited task scopes, insufficient evaluation dimensions, and heavy reliance on manual annotations, which significantly constrain their scalability and practical applicability. To address this, we propose \textbf{I2I-Bench}, a comprehensive benchmark for image-to-image editing models, which features (i) diverse tasks, encompassing 10 task categories across both single-image and multi-image editing tasks, (ii) comprehensive evaluation dimensions, including 30 decoupled and fine-grained evaluation dimensions with automated hybrid evaluation methods that integrate specialized tools and large multimodal models (LMMs), and (iii) rigorous alignment validation, justifying the consistency between our benchmark evaluations and human preferences. Using I2I-Bench, we benchmark numerous mainstream image editing models, investigating the gaps and trade-offs between editing models across various dimensions. We will open-source all components of I2I-Bench to facilitate future research.