Event-Based Video Frame Interpolation With Cross-Modal Asymmetric Bidirectional Motion Fields

TL;DR

This paper introduces a novel event-based video frame interpolation framework that estimates asymmetric bidirectional motion fields using both event data and images, achieving superior results on a new large-scale dataset.

Contribution

The paper proposes EIF-BiOFNet, a new framework for event-based VFI that directly estimates complex motion fields without approximation, and introduces ERF-X170FPS, a large-scale dataset for evaluation.

Findings

Significant performance improvements over state-of-the-art methods.

Effective utilization of event and image data for motion estimation.

Validation on a new high-quality, large-scale dataset.

Abstract

Video Frame Interpolation (VFI) aims to generate intermediate video frames between consecutive input frames. Since the event cameras are bio-inspired sensors that only encode brightness changes with a micro-second temporal resolution, several works utilized the event camera to enhance the performance of VFI. However, existing methods estimate bidirectional inter-frame motion fields with only events or approximations, which can not consider the complex motion in real-world scenarios. In this paper, we propose a novel event-based VFI framework with cross-modal asymmetric bidirectional motion field estimation. In detail, our EIF-BiOFNet utilizes each valuable characteristic of the events and images for direct estimation of inter-frame motion fields without any approximation methods. Moreover, we develop an interactive attention-based frame synthesis network to efficiently leverage the complementary warping-based and synthesis-based features. Finally, we build a large-scale event-based VFI dataset, ERF-X170FPS, with a high frame rate, extreme motion, and dynamic textures to overcome the limitations of previous event-based VFI datasets. Extensive experimental results validate that our method shows significant performance improvement over the state-of-the-art VFI methods on various datasets. Our project pages are available at: https://github.com/intelpro/CBMNet