Event-based Continuous Color Video Decompression from Single Frames

TL;DR

ContinuityCam reconstructs continuous color videos from a single image and event stream, leveraging motion modeling and neural synthesis to handle high-speed motion, lighting changes, and reduce bandwidth.

Contribution

It introduces a novel method combining motion modeling and neural synthesis for event-based video decompression from a single image, along with a new dataset and a specialized imaging setup.

Findings

Outperforms baseline methods by 3.61 dB in PSNR

Achieves 33% reduction in LPIPS

Demonstrates superior results on downstream tasks

Abstract

We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image and an event camera stream. Conventional cameras struggle with high-speed motion capture due to bandwidth and dynamic range limitations. Event cameras are ideal sensors to solve this problem because they encode compressed change information at high temporal resolution. In this work, we tackle the problem of event-based continuous color video decompression, pairing single static color frames and event data to reconstruct temporally continuous videos. Our approach combines continuous long-range motion modeling with a neural synthesis model, enabling frame prediction at arbitrary times within the events. Our method only requires an initial image, thus increasing the robustness to sudden motions, light changes, minimizing the prediction latency, and decreasing bandwidth usage. We also introduce a novel single-lens beamsplitter setup that acquires aligned images and events, and a novel and challenging Event Extreme Decompression Dataset (E2D2) that tests the method in various lighting and motion profiles. We thoroughly evaluate our method by benchmarking color frame reconstruction, outperforming the baseline methods by 3.61 dB in PSNR and by 33% decrease in LPIPS, as well as showing superior results on two downstream tasks.