Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction

TL;DR

This paper introduces a geometry-based quadratic model and a 3D video architecture to improve rolling shutter correction in complex, occluded, and nonlinear motion scenes, outperforming existing methods.

Contribution

It proposes a novel high-order correction field estimation and a multi-frame aggregation framework for better image quality in challenging dynamic scenes.

Findings

Surpasses state-of-the-art PSNR metrics on multiple datasets.

Accurately models pixel trajectories with a quadratic motion solver.

Effectively reconstructs occlusion frames in dynamic scenes.

Abstract

This paper addresses the problem of rolling shutter correction in complex nonlinear and dynamic scenes with extreme occlusion. Existing methods suffer from two main drawbacks. Firstly, they face challenges in estimating the accurate correction field due to the uniform velocity assumption, leading to significant image correction errors under complex motion. Secondly, the drastic occlusion in dynamic scenes prevents current solutions from achieving better image quality because of the inherent difficulties in aligning and aggregating multiple frames. To tackle these challenges, we model the curvilinear trajectory of pixels analytically and propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels. Besides, to reconstruct high-quality occlusion frames in dynamic scenes, we present a 3D video architecture that effectively Aligns and Aggregates multi-frame context, namely, RSA2-Net. We evaluate our method across a broad range of cameras and video sequences, demonstrating its significant superiority. Specifically, our method surpasses the state-of-the-art by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively. Code is available at https://github.com/DelinQu/qrsc.