WAIT: Feature Warping for Animation to Illustration video Translation using GANs

TL;DR

This paper introduces WAIT, a novel GAN-based method for stylizing videos from unordered sets of images, ensuring temporal consistency without relying on traditional optical flow or temporal predictors.

Contribution

We propose a new generator with feature warping layers that maintains temporal coherence in video stylization from unordered image sets, simplifying and speeding up the process.

Findings

Effective in stylizing animation videos with illustrations

Outperforms existing methods in qualitative and quantitative evaluations

Works well across multiple datasets

Abstract

In this paper, we explore a new domain for video-to-video translation. Motivated by the availability of animation movies that are adopted from illustrated books for children, we aim to stylize these videos with the style of the original illustrations. Current state-of-the-art video-to-video translation models rely on having a video sequence or a single style image to stylize an input video. We introduce a new problem for video stylizing where an unordered set of images are used. This is a challenging task for two reasons: i) we do not have the advantage of temporal consistency as in video sequences; ii) it is more difficult to obtain consistent styles for video frames from a set of unordered images compared to using a single image. Most of the video-to-video translation methods are built on an image-to-image translation model, and integrate additional networks such as optical flow, or temporal predictors to capture temporal relations. These additional networks make the model training and inference complicated and slow down the process. To ensure temporal coherency in video-to-video style transfer, we propose a new generator network with feature warping layers which overcomes the limitations of the previous methods. We show the effectiveness of our method on three datasets both qualitatively and quantitatively. Code and pretrained models are available at https://github.com/giddyyupp/wait.