Perception-Oriented Video Frame Interpolation via Asymmetric Blending

TL;DR

PerVFI introduces an asymmetric blending approach with a sparse mask and flow-based generator to enhance perceptual quality in video frame interpolation, effectively reducing blur and ghosting artifacts.

Contribution

The paper proposes a novel perception-oriented VFI framework with asymmetric blending, a sparse mask, and a flow-based generator, addressing motion errors and misalignment issues.

Findings

Significant perceptual quality improvements over existing methods

Effective reduction of ghosting and blur artifacts

Enhanced detail preservation in interpolated frames

Abstract

Previous methods for Video Frame Interpolation (VFI) have encountered challenges, notably the manifestation of blur and ghosting effects. These issues can be traced back to two pivotal factors: unavoidable motion errors and misalignment in supervision. In practice, motion estimates often prove to be error-prone, resulting in misaligned features. Furthermore, the reconstruction loss tends to bring blurry results, particularly in misaligned regions. To mitigate these challenges, we propose a new paradigm called PerVFI (Perception-oriented Video Frame Interpolation). Our approach incorporates an Asymmetric Synergistic Blending module (ASB) that utilizes features from both sides to synergistically blend intermediate features. One reference frame emphasizes primary content, while the other contributes complementary information. To impose a stringent constraint on the blending process, we introduce a self-learned sparse quasi-binary mask which effectively mitigates ghosting and blur artifacts in the output. Additionally, we employ a normalizing flow-based generator and utilize the negative log-likelihood loss to learn the conditional distribution of the output, which further facilitates the generation of clear and fine details. Experimental results validate the superiority of PerVFI, demonstrating significant improvements in perceptual quality compared to existing methods. Codes are available at \url{https://github.com/mulns/PerVFI}