SelfDRSC++: Self-Supervised Learning for Dual Reversed Rolling Shutter Correction

TL;DR

SelfDRSC++ introduces a self-supervised framework for correcting rolling shutter distortion in images, leveraging cycle consistency and a novel network architecture to improve performance and enable high framerate video generation.

Contribution

It proposes a lightweight, self-supervised dual reversed RS correction network with cycle consistency training, reducing reliance on ground-truth global shutter images and enabling high framerate video output.

Findings

Achieves superior correction performance with fewer parameters.

Enables training without high framerate GS ground-truth images.

Supports supervision at arbitrary intermediate scanning times.

Abstract

Modern consumer cameras commonly employ the rolling shutter (RS) imaging mechanism, via which images are captured by scanning scenes row-by-row, resulting in RS distortion for dynamic scenes. To correct RS distortion, existing methods adopt a fully supervised learning manner that requires high framerate global shutter (GS) images as ground-truth for supervision. In this paper, we propose an enhanced Self-supervised learning framework for Dual reversed RS distortion Correction (SelfDRSC++). Firstly, we introduce a lightweight DRSC network that incorporates a bidirectional correlation matching block to refine the joint optimization of optical flows and corrected RS features, thereby improving correction performance while reducing network parameters. Subsequently, to effectively train the DRSC network, we propose a self-supervised learning strategy that ensures cycle consistency between input and reconstructed dual reversed RS images. The RS reconstruction in SelfDRSC++ can be interestingly formulated as a specialized instance of video frame interpolation, where each row in reconstructed RS images is interpolated from predicted GS images by utilizing RS distortion time maps. By achieving superior performance while simplifying the training process, SelfDRSC++ enables feasible one-stage self-supervised training. Additionally, besides start and end RS scanning time, SelfDRSC++ allows supervision of GS images at arbitrary intermediate scanning times, thus enabling the learned DRSC network to generate high framerate GS videos. The code and trained models are available at \url{https://github.com/shangwei5/SelfDRSC_plusplus}.