The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting
Ryan Szeto, Jason J. Corso

TL;DR
The paper introduces DEVIL, a benchmark with a new dataset and evaluation scheme for video inpainting, focusing on failure modes and content attributes to better diagnose method performance.
Contribution
It presents a novel dataset with labeled failure modes and an evaluation scheme that isolates content attributes, enabling detailed analysis of video inpainting methods.
Findings
Reveals how specific content attributes affect inpainting performance.
Provides a diagnostic tool for systematic analysis of failure modes.
Enables targeted improvements in video inpainting techniques.
Abstract
Quantitative evaluation has increased dramatically among recent video inpainting work, but the video and mask content used to gauge performance has received relatively little attention. Although attributes such as camera and background scene motion inherently change the difficulty of the task and affect methods differently, existing evaluation schemes fail to control for them, thereby providing minimal insight into inpainting failure modes. To address this gap, we propose the Diagnostic Evaluation of Video Inpainting on Landscapes (DEVIL) benchmark, which consists of two contributions: (i) a novel dataset of videos and masks labeled according to several key inpainting failure modes, and (ii) an evaluation scheme that samples slices of the dataset characterized by a fixed content attribute, and scores performance on each slice according to reconstruction, realism, and temporal…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Image Processing Techniques · Cell Image Analysis Techniques
MethodsInpainting
