Deep Video Prior for Video Consistency and Propagation

TL;DR

This paper introduces a novel Deep Video Prior approach that achieves blind video temporal consistency and propagation by training a neural network directly on video pairs, outperforming existing methods in various tasks.

Contribution

The paper presents a new Deep Video Prior method trained on video pairs without large datasets, and extends it to effective video propagation with a progressive strategy.

Findings

Outperforms state-of-the-art in blind video temporal consistency

Effective in propagating color, style, and segmentation information

Uses iterative reweighted training to handle multimodal inconsistency

Abstract

Applying an image processing algorithm independently to each video frame often leads to temporal inconsistency in the resulting video. To address this issue, we present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. Unlike most previous methods that enforce temporal consistency with optical flow, we show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior (DVP). Moreover, a carefully designed iteratively reweighted training strategy is proposed to address the challenging multimodal inconsistency problem. We demonstrate the effectiveness of our approach on 7 computer vision tasks on videos. Extensive quantitative and perceptual experiments show that our approach obtains superior performance than state-of-the-art methods on blind video temporal consistency. We further extend DVP to video propagation and demonstrate its effectiveness in propagating three different types of information (color, artistic style, and object segmentation). A progressive propagation strategy with pseudo labels is also proposed to enhance DVP's performance on video propagation. Our source codes are publicly available at https://github.com/ChenyangLEI/deep-video-prior.