Enhanced Deep Animation Video Interpolation

TL;DR

This paper introduces AutoFI, a method for generating synthetic training data tailored for animation video interpolation, and SktFI, a sketch-based post-processing module to improve frame quality, addressing challenges in cartoon video interpolation.

Contribution

The paper presents AutoFI for automatic synthetic data generation and SktFI for manual refinement, enhancing deep animation video interpolation methods.

Findings

AutoFI improves training for DAIN and ANIN on cartoon videos.

SktFI effectively refines frames with user sketches.

High perceptual quality achieved in interpolated animation frames.

Abstract

Existing learning-based frame interpolation algorithms extract consecutive frames from high-speed natural videos to train the model. Compared to natural videos, cartoon videos are usually in a low frame rate. Besides, the motion between consecutive cartoon frames is typically nonlinear, which breaks the linear motion assumption of interpolation algorithms. Thus, it is unsuitable for generating a training set directly from cartoon videos. For better adapting frame interpolation algorithms from nature video to animation video, we present AutoFI, a simple and effective method to automatically render training data for deep animation video interpolation. AutoFI takes a layered architecture to render synthetic data, which ensures the assumption of linear motion. Experimental results show that AutoFI performs favorably in training both DAIN and ANIN. However, most frame interpolation algorithms will still fail in error-prone areas, such as fast motion or large occlusion. Besides AutoFI, we also propose a plug-and-play sketch-based post-processing module, named SktFI, to refine the final results using user-provided sketches manually. With AutoFI and SktFI, the interpolated animation frames show high perceptual quality.