Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events

TL;DR

This paper introduces a self-supervised, event-based neural network for recovering high-quality global shutter videos from rolling shutter images, leveraging event cameras' high temporal resolution to overcome the ill-posed nature of the problem.

Contribution

It proposes a novel event-based SDR network with a self-supervised learning paradigm, including an Event-based Inter/intra-frame Compensator (E-IC) for dynamic prediction without ground-truth images.

Findings

Achieves state-of-the-art performance on synthetic and real datasets.

Effectively reconstructs undistorted high frame-rate videos from rolling shutter images.

Demonstrates robustness in real-world scenarios.

Abstract

Scene Dynamic Recovery (SDR) by inverting distorted Rolling Shutter (RS) images to an undistorted high frame-rate Global Shutter (GS) video is a severely ill-posed problem due to the missing temporal dynamic information in both RS intra-frame scanlines and inter-frame exposures, particularly when prior knowledge about camera/object motions is unavailable. Commonly used artificial assumptions on scenes/motions and data-specific characteristics are prone to producing sub-optimal solutions in real-world scenarios. To address this challenge, we propose an event-based SDR network within a self-supervised learning paradigm, i.e., SelfUnroll. We leverage the extremely high temporal resolution of event cameras to provide accurate inter/intra-frame dynamic information. Specifically, an Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals, including the temporal transition and spatial translation. Exploring connections in terms of RS-RS, RS-GS, and GS-RS, we explicitly formulate mutual constraints with the proposed E-IC, resulting in supervisions without ground-truth GS images. Extensive evaluations over synthetic and real datasets demonstrate that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios. The dataset and code are available at https://w3un.github.io/selfunroll/.