# Rolling-Shutter-Aware Differential SfM and Image Rectification

## TL;DR

This paper introduces a novel RS-aware differential SfM method that estimates relative pose and performs high-quality image rectification despite rolling shutter distortions, outperforming existing solutions.

## Contribution

It presents a linear scaling correction for RS artifacts under constant velocity and a 9-point algorithm for acceleration, improving pose estimation and image rectification accuracy.

## Key findings

- More accurate relative pose estimation on RS images
- Enhanced 3D reconstruction quality from RS images
- Superior image rectification compared to commercial software

## Abstract

In this paper, we develop a modified differential Structure from Motion (SfM) algorithm that can estimate relative pose from two consecutive frames despite of Rolling Shutter (RS) artifacts. In particular, we show that under constant velocity assumption, the errors induced by the rolling shutter effect can be easily rectified by a linear scaling operation on each optical flow. We further propose a 9-point algorithm to recover the relative pose of a rolling shutter camera that undergoes constant acceleration motion. We demonstrate that the dense depth maps recovered from the relative pose of the RS camera can be used in a RS-aware warping for image rectification to recover high-quality Global Shutter (GS) images. Experiments on both synthetic and real RS images show that our RS-aware differential SfM algorithm produces more accurate results on relative pose estimation and 3D reconstruction from images distorted by RS effect compared to standard SfM algorithms that assume a GS camera model. We also demonstrate that our RS-aware warping for image rectification method outperforms state-of-the-art commercial software products, i.e. Adobe After Effects and Apple Imovie, at removing RS artifacts.