Exploring Discontinuity for Video Frame Interpolation

TL;DR

This paper introduces techniques to improve video frame interpolation models' robustness to unnatural objects with discontinuous motions, using data augmentation, a discontinuity map, and specialized loss functions, validated on a new benchmark and existing datasets.

Contribution

It presents a novel data augmentation method, a discontinuity map prediction module, and loss functions to enhance VFI models' performance on videos with discontinuous motions.

Findings

Significant quality improvements on GDM dataset.

Enhanced performance on existing benchmarks like Vimeo90K, UCF101, and DAVIS.

Effective handling of unnatural objects with discontinuous motions.

Abstract

Video frame interpolation (VFI) is the task that synthesizes the intermediate frame given two consecutive frames. Most of the previous studies have focused on appropriate frame warping operations and refinement modules for the warped frames. These studies have been conducted on natural videos containing only continuous motions. However, many practical videos contain various unnatural objects with discontinuous motions such as logos, user interfaces and subtitles. We propose three techniques to make the existing deep learning-based VFI architectures robust to these elements. First is a novel data augmentation strategy called figure-text mixing (FTM) which can make the models learn discontinuous motions during training stage without any extra dataset. Second, we propose a simple but effective module that predicts a map called discontinuity map (D-map), which densely distinguishes between areas of continuous and discontinuous motions. Lastly, we propose loss functions to give supervisions of the discontinuous motion areas which can be applied along with FTM and D-map. We additionally collect a special test benchmark called Graphical Discontinuous Motion (GDM) dataset consisting of some mobile games and chatting videos. Applied to the various state-of-the-art VFI networks, our method significantly improves the interpolation qualities on the videos from not only GDM dataset, but also the existing benchmarks containing only continuous motions such as Vimeo90K, UCF101, and DAVIS.