Event-based Video Frame Interpolation with Edge Guided Motion Refinement

TL;DR

This paper presents an end-to-end event-based video frame interpolation method that leverages edge features from event signals for improved motion estimation and warping, enhancing interpolation quality.

Contribution

The proposed EGMR method introduces an edge-guided attentive module and a learned visibility map to better utilize event data for motion refinement in video frame interpolation.

Findings

Outperforms existing methods on synthetic and real datasets.

Effectively utilizes edge features for motion flow correction.

Reduces occlusion artifacts in interpolated frames.

Abstract

Video frame interpolation, the process of synthesizing intermediate frames between sequential video frames, has made remarkable progress with the use of event cameras. These sensors, with microsecond-level temporal resolution, fill information gaps between frames by providing precise motion cues. However, contemporary Event-Based Video Frame Interpolation (E-VFI) techniques often neglect the fact that event data primarily supply high-confidence features at scene edges during multi-modal feature fusion, thereby diminishing the role of event signals in optical flow (OF) estimation and warping refinement. To address this overlooked aspect, we introduce an end-to-end E-VFI learning method (referred to as EGMR) to efficiently utilize edge features from event signals for motion flow and warping enhancement. Our method incorporates an Edge Guided Attentive (EGA) module, which rectifies estimated video motion through attentive aggregation based on the local correlation of multi-modal features in a coarse-to-fine strategy. Moreover, given that event data can provide accurate visual references at scene edges between consecutive frames, we introduce a learned visibility map derived from event data to adaptively mitigate the occlusion problem in the warping refinement process. Extensive experiments on both synthetic and real datasets show the effectiveness of the proposed approach, demonstrating its potential for higher quality video frame interpolation.