Learning Long-Term Style-Preserving Blind Video Temporal Consistency

TL;DR

This paper introduces a recurrent neural network model that enhances long-term temporal consistency in videos while preserving style, outperforming existing methods in flicker removal and style retention.

Contribution

A novel style-preserving perceptual loss and a Ping Pong training procedure enable effective long-term consistency and style preservation in video post-processing.

Findings

State-of-the-art flicker removal on DAVIS and videvo.net datasets.

Better style preservation compared to previous methods.

Effective long-term temporal consistency achieved.

Abstract

When trying to independently apply image-trained algorithms to successive frames in videos, noxious flickering tends to appear. State-of-the-art post-processing techniques that aim at fostering temporal consistency, generate other temporal artifacts and visually alter the style of videos. We propose a postprocessing model, agnostic to the transformation applied to videos (e.g. style transfer, image manipulation using GANs, etc.), in the form of a recurrent neural network. Our model is trained using a Ping Pong procedure and its corresponding loss, recently introduced for GAN video generation, as well as a novel style preserving perceptual loss. The former improves long-term temporal consistency learning, while the latter fosters style preservation. We evaluate our model on the DAVIS and videvo.net datasets and show that our approach offers state-of-the-art results concerning flicker removal, and better keeps the overall style of the videos than previous approaches.