DeepOIS: Gyroscope-Guided Deep Optical Image Stabilizer Compensation

TL;DR

DeepOIS introduces a deep learning approach that uses gyroscope data to compensate for optical image stabilizer effects, enabling accurate image alignment solely based on sensor data, thus improving robustness and performance.

Contribution

The paper presents a novel deep network that leverages gyroscope data to correct OIS-induced motions, allowing gyroscope-based image alignment in stabilized cameras.

Findings

Achieves image alignment comparable to non-OIS cameras.

Outperforms traditional image-based alignment methods significantly.

Provides a new dataset and code for further research.

Abstract

Mobile captured images can be aligned using their gyroscope sensors. Optical image stabilizer (OIS) terminates this possibility by adjusting the images during the capturing. In this work, we propose a deep network that compensates the motions caused by the OIS, such that the gyroscopes can be used for image alignment on the OIS cameras. To achieve this, first, we record both videos and gyroscopes with an OIS camera as training data. Then, we convert gyroscope readings into motion fields. Second, we propose a Fundamental Mixtures motion model for rolling shutter cameras, where an array of rotations within a frame are extracted as the ground-truth guidance. Third, we train a convolutional neural network with gyroscope motions as input to compensate for the OIS motion. Once finished, the compensation network can be applied for other scenes, where the image alignment is purely based on gyroscopes with no need for images contents, delivering strong robustness. Experiments show that our results are comparable with that of non-OIS cameras, and outperform image-based alignment results with a relatively large margin. Code and dataset are available at https://github.com/lhaippp/DeepOIS