Video Frame Interpolation with Densely Queried Bilateral Correlation

TL;DR

This paper introduces Densely Queried Bilateral Correlation (DQBC), a novel method for video frame interpolation that improves motion estimation accuracy, especially for small and fast-moving objects, leading to better synthesized frames.

Contribution

The paper proposes DQBC, a new correlation modeling technique that eliminates receptive field dependency, enhancing motion estimation for challenging small and fast-moving objects in VFI.

Findings

Higher accuracy than state-of-the-art methods

Reduced inference time

Better handling of small and fast-moving objects

Abstract

Video Frame Interpolation (VFI) aims to synthesize non-existent intermediate frames between existent frames. Flow-based VFI algorithms estimate intermediate motion fields to warp the existent frames. Real-world motions' complexity and the reference frame's absence make motion estimation challenging. Many state-of-the-art approaches explicitly model the correlations between two neighboring frames for more accurate motion estimation. In common approaches, the receptive field of correlation modeling at higher resolution depends on the motion fields estimated beforehand. Such receptive field dependency makes common motion estimation approaches poor at coping with small and fast-moving objects. To better model correlations and to produce more accurate motion fields, we propose the Densely Queried Bilateral Correlation (DQBC) that gets rid of the receptive field dependency problem and thus is more friendly to small and fast-moving objects. The motion fields generated with the help of DQBC are further refined and up-sampled with context features. After the motion fields are fixed, a CNN-based SynthNet synthesizes the final interpolated frame. Experiments show that our approach enjoys higher accuracy and less inference time than the state-of-the-art. Source code is available at https://github.com/kinoud/DQBC.