RealDrag: The First Dragging Benchmark with Real Target Image

TL;DR

RealDrag introduces the first standardized benchmark dataset with paired ground truth images and novel metrics for evaluating point-based image editing models, enabling fairer and more consistent comparisons.

Contribution

It provides a comprehensive dataset with ground truth images, diverse samples, and four new evaluation metrics for point-based image editing.

Findings

Evaluated 17 state-of-the-art models systematically.

Identified trade-offs among current approaches.

Established a reproducible baseline for future research.

Abstract

The evaluation of drag based image editing models is unreliable due to a lack of standardized benchmarks and metrics. This ambiguity stems from inconsistent evaluation protocols and, critically, the absence of datasets containing ground truth target images, making objective comparisons between competing methods difficult. To address this, we introduce \textbf{RealDrag}, the first comprehensive benchmark for point based image editing that includes paired ground truth target images. Our dataset contains over 400 human annotated samples from diverse video sources, providing source/target images, handle/target points, editable region masks, and descriptive captions for both the image and the editing action. We also propose four novel, task specific metrics: Semantical Distance (SeD), Outer Mask Preserving Score (OMPS), Inner Patch Preserving Score (IPPS), and Directional Similarity (DiS). These metrics are designed to quantify pixel level matching fidelity, check preservation of non edited (out of mask) regions, and measure semantic alignment with the desired task. Using this benchmark, we conduct the first large scale systematic analysis of the field, evaluating 17 SOTA models. Our results reveal clear trade offs among current approaches and establish a robust, reproducible baseline to guide future research. Our dataset and evaluation toolkit will be made publicly available.