PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media

TL;DR

PROVE introduces perception-aligned metrics RC-S and RC-T, along with PROVE-Bench, a comprehensive benchmark, to improve evaluation of object removal in images and videos by better aligning with human perception.

Contribution

The paper presents RC metrics and PROVE-Bench, addressing limitations of existing metrics and establishing a new evaluation framework for visual media object removal.

Findings

RC metrics outperform existing metrics in aligning with human judgments.

PROVE-Bench provides a challenging real-world dataset for benchmarking.

Experiments validate the effectiveness of RC metrics across diverse benchmarks.

Abstract

Evaluating object removal in images and videos remains challenging because the task is inherently one-to-many, yet existing metrics frequently disagree with human perception. Full-reference metrics reward copy-paste behaviors over genuine erasure; no-reference metrics suffer from systematic biases such as favoring blurry results; and global temporal metrics are insensitive to localized artifacts within edited regions. To address these limitations, we propose RC (Removal Coherence), a pair of perception-aligned metrics: RC-S, which measures spatial coherence via sliding-window feature comparison between masked and background regions, and RC-T, which measures temporal consistency via distribution tracking within shared restored regions across adjacent frames. To validate RC and support community benchmarking, we further introduce PROVE-Bench, a two-tier real-world benchmark comprising PROVE-M, an 80-video paired dataset with motion augmentation, and PROVE-H, a 100-video challenging subset without ground truth. Together, RC metrics and PROVE-Bench form the PROVE (Perceptual RemOVal cohErence) evaluation framework for visual media. Experiments across diverse image and video benchmarks demonstrate that RC achieves substantially stronger alignment with human judgments than existing evaluation protocols. The code for RC metrics and PROVE-Bench are publicly available at: https://github.com/xiaomi-research/prove/.