TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation

TL;DR

TimeReplayer leverages event camera data to enable high-quality video interpolation from low-FPS videos without high-speed training data, effectively modeling complex motions and allowing video extrapolation.

Contribution

The paper introduces a novel unsupervised cycle-consistent algorithm, TimeReplayer, that utilizes event camera data for video interpolation, eliminating the need for costly high-speed training data.

Findings

Achieves state-of-the-art video interpolation results.

Supports video extrapolation beyond original frames.

Demonstrates promising potential of event-based vision.

Abstract

Recording fast motion in a high FPS (frame-per-second) requires expensive high-speed cameras. As an alternative, interpolating low-FPS videos from commodity cameras has attracted significant attention. If only low-FPS videos are available, motion assumptions (linear or quadratic) are necessary to infer intermediate frames, which fail to model complex motions. Event camera, a new camera with pixels producing events of brightness change at the temporal resolution of $\mu s$ $(10^{-6}$ second $)$, is a game-changing device to enable video interpolation at the presence of arbitrarily complex motion. Since event camera is a novel sensor, its potential has not been fulfilled due to the lack of processing algorithms. The pioneering work Time Lens introduced event cameras to video interpolation by designing optical devices to collect a large amount of paired training data of high-speed frames and events, which is too costly to scale. To fully unlock the potential of event cameras, this paper proposes a novel TimeReplayer algorithm to interpolate videos captured by commodity cameras with events. It is trained in an unsupervised cycle-consistent style, canceling the necessity of high-speed training data and bringing the additional ability of video extrapolation. Its state-of-the-art results and demo videos in supplementary reveal the promising future of event-based vision.