Self-Supervised Real-time Video Stabilization

TL;DR

This paper introduces a self-supervised, real-time video stabilization method that minimizes distortion and cropping, outperforming existing techniques and operating at 41 fps across various resolutions.

Contribution

A novel self-supervised framework for real-time video stabilization that eliminates the need for special hardware and reduces post-processing artifacts.

Findings

Outperforms state-of-the-art real-time stabilization methods.

Operates at 41 fps regardless of resolution.

Reduces distortion and margin cropping effectively.

Abstract

Videos are a popular media form, where online video streaming has recently gathered much popularity. In this work, we propose a novel method of real-time video stabilization - transforming a shaky video to a stabilized video as if it were stabilized via gimbals in real-time. Our framework is trainable in a self-supervised manner, which does not require data captured with special hardware setups (i.e., two cameras on a stereo rig or additional motion sensors). Our framework consists of a transformation estimator between given frames for global stability adjustments, followed by scene parallax reduction module via spatially smoothed optical flow for further stability. Then, a margin inpainting module fills in the missing margin regions created during stabilization to reduce the amount of post-cropping. These sequential steps reduce distortion and margin cropping to a minimum while enhancing stability. Hence, our approach outperforms state-of-the-art real-time video stabilization methods as well as offline methods that require camera trajectory optimization. Our method procedure takes approximately 24.3 ms yielding 41 fps regardless of resolution (e.g., 480p or 1080p).