StableI2I: Spotting Unintended Changes in Image-to-Image Transition

TL;DR

StableI2I introduces a comprehensive evaluation framework for assessing content fidelity and spatial consistency in image-to-image tasks, addressing limitations of existing metrics that overlook semantic preservation.

Contribution

It proposes a unified, dynamic evaluation method and a benchmark for measuring content fidelity and consistency in I2I tasks without reference images.

Findings

StableI2I provides accurate and interpretable evaluations.

Strong correlation with human judgments.

Effective across diverse I2I tasks.

Abstract

In most real-world image-to-image (I2I) scenarios, existing evaluations primarily focus on instruction following and the perceptual quality or aesthetics of the generated images. However, they largely fail to assess whether the output image preserves the semantic correspondence and spatial structure of the input image. To address this limitation, we propose StableI2I, a unified and dynamic evaluation framework that explicitly measures content fidelity and pre--post consistency across a wide range of I2I tasks without requiring reference images, including image editing and image restoration. In addition, we construct StableI2I-Bench, a benchmark designed to systematically evaluate the accuracy of MLLMs on such fidelity and consistency assessment tasks. Extensive experimental results demonstrate that StableI2I provides accurate, fine-grained, and interpretable evaluations of content fidelity and consistency, with strong correlations to human subjective judgments. Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.