Trusted Video Inpainting Localization via Deep Attentive Noise Learning

TL;DR

This paper introduces TruVIL, a deep learning-based method for reliably localizing inpainted regions in videos by leveraging high-frequency noise analysis and multi-stage attentive feature fusion, with a new annotated dataset.

Contribution

The paper proposes a novel deep attentive noise learning framework for video inpainting localization and provides a large annotated dataset for training and evaluation.

Findings

TruVIL outperforms existing methods in localization accuracy.

High-frequency noise effectively reveals inpainted regions.

The method demonstrates strong robustness and generalization across various videos.

Abstract

Digital video inpainting techniques have been substantially improved with deep learning in recent years. Although inpainting is originally designed to repair damaged areas, it can also be used as malicious manipulation to remove important objects for creating false scenes and facts. As such it is significant to identify inpainted regions blindly. In this paper, we present a Trusted Video Inpainting Localization network (TruVIL) with excellent robustness and generalization ability. Observing that high-frequency noise can effectively unveil the inpainted regions, we design deep attentive noise learning in multiple stages to capture the inpainting traces. Firstly, a multi-scale noise extraction module based on 3D High Pass (HP3D) layers is used to create the noise modality from input RGB frames. Then the correlation between such two complementary modalities are explored by a cross-modality attentive fusion module to facilitate mutual feature learning. Lastly, spatial details are selectively enhanced by an attentive noise decoding module to boost the localization performance of the network. To prepare enough training samples, we also build a frame-level video object segmentation dataset of 2500 videos with pixel-level annotation for all frames. Extensive experimental results validate the superiority of TruVIL compared with the state-of-the-arts. In particular, both quantitative and qualitative evaluations on various inpainted videos verify the remarkable robustness and generalization ability of our proposed TruVIL. Code and dataset will be available at https://github.com/multimediaFor/TruVIL.