Learning Blind Video Temporal Consistency

TL;DR

This paper introduces a deep recurrent network that enforces temporal consistency across video frames, working with various image processing tasks without task-specific domain knowledge, and operates in real-time.

Contribution

The proposed end-to-end deep recurrent network achieves real-time, task-agnostic temporal consistency in videos without optical flow, handling multiple unseen tasks effectively.

Findings

Outperforms state-of-the-art methods on various video tasks

Operates in real-time even on high-resolution videos

Handles multiple and unseen video processing tasks

Abstract

Applying image processing algorithms independently to each frame of a video often leads to undesired inconsistent results over time. Developing temporally consistent video-based extensions, however, requires domain knowledge for individual tasks and is unable to generalize to other applications. In this paper, we present an efficient end-to-end approach based on deep recurrent network for enforcing temporal consistency in a video. Our method takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video. Consequently, our approach is agnostic to specific image processing algorithms applied on the original video. We train the proposed network by minimizing both short-term and long-term temporal losses as well as the perceptual loss to strike a balance between temporal stability and perceptual similarity with the processed frames. At test time, our model does not require computing optical flow and thus achieves real-time speed even for high-resolution videos. We show that our single model can handle multiple and unseen tasks, including but not limited to artistic style transfer, enhancement, colorization, image-to-image translation and intrinsic image decomposition. Extensive objective evaluation and subject study demonstrate that the proposed approach performs favorably against the state-of-the-art methods on various types of videos.