Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames

TL;DR

This paper introduces a self-supervised framework that combines event cameras and rolling shutter images to produce high-quality, high-frame-rate global shutter videos free of distortion, significantly reducing bandwidth needs.

Contribution

It presents a novel self-supervised method for reconstructing global shutter videos from rolling shutter and event data, enabling distortion removal and slow-motion recovery.

Findings

Reduces bandwidth by 94%

Achieves 16 ms per frame at 32x interpolation

Removes distortion effectively

Abstract

Most consumer cameras use rolling shutter (RS) exposure, which often leads to distortions such as skew and jelly effects. These videos are further limited by bandwidth and frame rate constraints. In this paper, we explore the potential of event cameras, which offer high temporal resolution. We propose a framework to recover global shutter (GS) high-frame-rate videos without RS distortion by combining an RS camera and an event camera. Due to the lack of real-world datasets, our framework adopts a self-supervised strategy based on a displacement field, a dense 3D spatiotemporal representation of pixel motion during exposure. This enables mutual reconstruction between RS and GS frames and facilitates slow-motion recovery. We combine RS frames with the displacement field to generate GS frames, and integrate inverse mapping and RS frame warping for self-supervision. Experiments on four datasets show that our method removes distortion, reduces bandwidth usage by 94 percent, and achieves 16 ms per frame at 32x interpolation.