DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting

TL;DR

DeMFI-Net is a novel framework that jointly performs video deblurring and multi-frame interpolation, achieving state-of-the-art results by effectively utilizing flow-guided attentive correlation and recursive boosting.

Contribution

The paper introduces DeMFI-Net, a joint deblurring and multi-frame interpolation model with a flow-guided attentive correlation module and recursive boosting, improving performance over prior separate approaches.

Findings

Achieves state-of-the-art performance on diverse datasets.

Effectively gathers blurry pixel information in feature space.

Significantly outperforms recent methods in both deblurring and interpolation.

Abstract

In this paper, we propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework, called DeMFI-Net, which accurately converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). The DeMFI-Net jointly performs deblurring and MFI where its baseline version performs feature-flow-based warping with FAC-FB module to obtain a sharp-interpolated frame as well to deblur two center-input frames. Moreover, its extended version further improves the joint task performance based on pixel-flow-based warping with GRU-based RB. Our FAC-FB module effectively gathers the distributed blurry pixel information over blurry input frames in feature-domain to improve the overall joint performances, which is computationally efficient since its attentive correlation is only focused pointwise. As a result, our DeMFI-Net achieves state-of-the-art (SOTA) performances for diverse datasets with significant margins compared to the recent SOTA methods, for both deblurring and MFI. All source codes including pretrained DeMFI-Net are publicly available at https://github.com/JihyongOh/DeMFI.