XVFI: eXtreme Video Frame Interpolation

TL;DR

This paper introduces XVFI-Net, a novel network for extreme video frame interpolation on 4K videos with large motion, supported by a new high-frame-rate dataset, achieving superior performance over existing methods.

Contribution

The paper presents a new dataset X4K1000FPS and a novel XVFI-Net architecture designed for high-quality interpolation of 4K videos with large motions.

Findings

XVFI-Net outperforms state-of-the-art methods on extreme motion videos.

The new dataset enables better training and evaluation of VFI models.

The framework is robust across different resolutions and motion complexities.

Abstract

In this paper, we firstly present a dataset (X4K1000FPS) of 4K videos of 1000 fps with the extreme motion to the research community for video frame interpolation (VFI), and propose an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. The XVFI-Net is based on a recursive multi-scale shared structure that consists of two cascaded modules for bidirectional optical flow learning between two input frames (BiOF-I) and for bidirectional optical flow learning from target to input frames (BiOF-T). The optical flows are stably approximated by a complementary flow reversal (CFR) proposed in BiOF-T module. During inference, the BiOF-I module can start at any scale of input while the BiOF-T module only operates at the original input scale so that the inference can be accelerated while maintaining highly accurate VFI performance. Extensive experimental results show that our XVFI-Net can successfully capture the essential information of objects with extremely large motions and complex textures while the state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows a robustness of our algorithm as well. All source codes, pre-trained models, and proposed X4K1000FPS datasets are publicly available at https://github.com/JihyongOh/XVFI.