IMU-Assisted Learning of Single-View Rolling Shutter Correction

TL;DR

This paper introduces a deep learning method that uses IMU data to improve the correction of rolling shutter distortion in images, enhancing the accuracy of vision algorithms like SLAM.

Contribution

It integrates IMU data into a neural network for better depth and pose prediction, significantly improving rolling shutter correction accuracy.

Findings

Enhanced pose prediction accuracy with IMU integration

Improved SLAM performance on corrected images

Extended dataset with real rolling shutter data

Abstract

Rolling shutter distortion is highly undesirable for photography and computer vision algorithms (e.g., visual SLAM) because pixels can be potentially captured at different times and poses. In this paper, we propose a deep neural network to predict depth and row-wise pose from a single image for rolling shutter correction. Our contribution in this work is to incorporate inertial measurement unit (IMU) data into the pose refinement process, which, compared to the state-of-the-art, greatly enhances the pose prediction. The improved accuracy and robustness make it possible for numerous vision algorithms to use imagery captured by rolling shutter cameras and produce highly accurate results. We also extend a dataset to have real rolling shutter images, IMU data, depth maps, camera poses, and corresponding global shutter images for rolling shutter correction training. We demonstrate the efficacy of the proposed method by evaluating the performance of Direct Sparse Odometry (DSO) algorithm on rolling shutter imagery corrected using the proposed approach. Results show marked improvements of the DSO algorithm over using uncorrected imagery, validating the proposed approach.