Single Image Rolling Shutter Removal with Diffusion Models

TL;DR

RS-Diffusion introduces a diffusion model-based approach for correcting rolling shutter artifacts in single images, leveraging a new dataset and outperforming previous methods.

Contribution

The paper proposes the first diffusion model-based framework for single-image rolling shutter correction and introduces the RS-Real dataset with ground-truth GS frames.

Findings

RS-Diffusion outperforms previous single-frame RS correction methods.

The RS-Real dataset provides paired RS and GS images with IMU data.

Diffusion techniques show promise for image correction tasks.

Abstract

We present RS-Diffusion, the first Diffusion Models-based method for single-frame Rolling Shutter (RS) correction. RS artifacts compromise visual quality of frames due to the row-wise exposure of CMOS sensors. Most previous methods have focused on multi-frame approaches, using temporal information from consecutive frames for the motion rectification. However, few approaches address the more challenging but important single frame RS correction. In this work, we present an ``image-to-motion" framework via diffusion techniques, with a designed patch-attention module. In addition, we present the RS-Real dataset, comprised of captured RS frames alongside their corresponding Global Shutter (GS) ground-truth pairs. The GS frames are corrected from the RS ones, guided by the corresponding Inertial Measurement Unit (IMU) gyroscope data acquired during capture. Experiments show that RS-Diffusion surpasses previous single-frame RS methods, demonstrates the potential of diffusion-based approaches, and provides a valuable dataset for further research.