Revisiting Event-based Video Frame Interpolation

TL;DR

This paper introduces a novel event-guided optical flow refinement and incremental synthesis strategy for event-based video frame interpolation, improving accuracy by leveraging event characteristics and RGB data.

Contribution

It proposes a new approach that respects event stream properties, incorporating RGB info and incremental synthesis to enhance frame interpolation quality.

Findings

Improved interpolation accuracy over previous methods.

Event characteristics like high temporal density and noise influence results.

Incremental synthesis yields more realistic frames.

Abstract

Dynamic vision sensors or event cameras provide rich complementary information for video frame interpolation. Existing state-of-the-art methods follow the paradigm of combining both synthesis-based and warping networks. However, few of those methods fully respect the intrinsic characteristics of events streams. Given that event cameras only encode intensity changes and polarity rather than color intensities, estimating optical flow from events is arguably more difficult than from RGB information. We therefore propose to incorporate RGB information in an event-guided optical flow refinement strategy. Moreover, in light of the quasi-continuous nature of the time signals provided by event cameras, we propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages rather than in a single, long stage. Extensive experiments on both synthetic and real-world datasets show that these modifications lead to more reliable and realistic intermediate frame results than previous video frame interpolation methods. Our findings underline that a careful consideration of event characteristics such as high temporal density and elevated noise benefits interpolation accuracy.