Enhanced Correlation Matching based Video Frame Interpolation

TL;DR

This paper introduces a high-resolution video frame interpolation method using enhanced correlation matching and recurrent pyramid architecture, improving accuracy in challenging scenarios like large motion and occlusion.

Contribution

It presents a novel DNN framework with recursive flow refinement and correlation-based matching, optimized for 4K resolution and large motions, outperforming previous methods.

Findings

Outperforms previous methods on 4K video data

Achieves higher quality in low-resolution benchmarks

Uses fewer model parameters for comparable or better results

Abstract

We propose a novel DNN based framework called the Enhanced Correlation Matching based Video Frame Interpolation Network to support high resolution like 4K, which has a large scale of motion and occlusion. Considering the extensibility of the network model according to resolution, the proposed scheme employs the recurrent pyramid architecture that shares the parameters among each pyramid layer for optical flow estimation. In the proposed flow estimation, the optical flows are recursively refined by tracing the location with maximum correlation. The forward warping based correlation matching enables to improve the accuracy of flow update by excluding incorrectly warped features around the occlusion area. Based on the final bi-directional flows, the intermediate frame at arbitrary temporal position is synthesized using the warping and blending network and it is further improved by refinement network. Experiment results demonstrate that the proposed scheme outperforms the previous works at 4K video data and low-resolution benchmark datasets as well in terms of objective and subjective quality with the smallest number of model parameters.