Learning Adaptive Warping for Real-World Rolling Shutter Correction

TL;DR

This paper introduces a new real-world dataset for rolling shutter correction and proposes an adaptive warping model that effectively removes RS effects in dynamic scenes captured by consumer devices.

Contribution

The paper presents the first real-world RS correction dataset BS-RSC and a novel adaptive warping model for improved RS effect removal in real videos.

Findings

The dataset includes diverse real-world motions with ground truth frames.

The proposed model outperforms existing methods in correcting RS effects.

Experimental results validate the effectiveness of the dataset and the model.

Abstract

This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process, calling for RS effect removal techniques. However, current state-of-the-art RS correction methods often fail to remove RS effects in real scenarios since the motions are various and hard to model. To address this issue, we propose a real-world RS correction dataset BS-RSC. Real distorted videos with corresponding ground truth are recorded simultaneously via a well-designed beam-splitter-based acquisition system. BS-RSC contains various motions of both camera and objects in dynamic scenes. Further, an RS correction model with adaptive warping is proposed. Our model can warp the learned RS features into global shutter counterparts adaptively with predicted multiple displacement fields. These warped features are aggregated and then reconstructed into high-quality global shutter frames in a coarse-to-fine strategy. Experimental results demonstrate the effectiveness of the proposed method, and our dataset can improve the model's ability to remove the RS effects in the real world.