ReMOVE: A Reference-free Metric for Object Erasure

TL;DR

ReMOVE is a new reference-free metric designed to evaluate the effectiveness of object erasure in diffusion-based image editing, addressing the limitations of existing measures by better aligning with human perception.

Contribution

It introduces ReMOVE, a novel evaluation metric that accurately assesses object erasure quality without requiring a reference image, improving upon existing metrics.

Findings

ReMOVE correlates well with human judgments.

It effectively distinguishes between object removal and replacement.

ReMOVE aligns with state-of-the-art metrics.

Abstract

We introduce $\texttt{ReMOVE}$, a novel reference-free metric for assessing object erasure efficacy in diffusion-based image editing models post-generation. Unlike existing measures such as LPIPS and CLIPScore, $\texttt{ReMOVE}$ addresses the challenge of evaluating inpainting without a reference image, common in practical scenarios. It effectively distinguishes between object removal and replacement. This is a key issue in diffusion models due to stochastic nature of image generation. Traditional metrics fail to align with the intuitive definition of inpainting, which aims for (1) seamless object removal within masked regions (2) while preserving the background continuity. $\texttt{ReMOVE}$ not only correlates with state-of-the-art metrics and aligns with human perception but also captures the nuanced aspects of the inpainting process, providing a finer-grained evaluation of the generated outputs.