Minimum Latency Deep Online Video Stabilization

TL;DR

This paper introduces a new online video stabilization method that optimizes camera paths using deep motion models and a hybrid loss, significantly improving stabilization quality in real-time applications.

Contribution

The paper proposes a novel camera path optimization framework for online video stabilization that leverages deep motion models and a hybrid loss for improved spatial and temporal consistency.

Findings

Outperforms state-of-the-art online stabilization methods.

Achieves comparable results to offline methods.

Provides a new dataset for training and evaluation.

Abstract

We present a novel camera path optimization framework for the task of online video stabilization. Typically, a stabilization pipeline consists of three steps: motion estimating, path smoothing, and novel view rendering. Most previous methods concentrate on motion estimation, proposing various global or local motion models. In contrast, path optimization receives relatively less attention, especially in the important online setting, where no future frames are available. In this work, we adopt recent off-the-shelf high-quality deep motion models for motion estimation to recover the camera trajectory and focus on the latter two steps. Our network takes a short 2D camera path in a sliding window as input and outputs the stabilizing warp field of the last frame in the window, which warps the coming frame to its stabilized position. A hybrid loss is well-defined to constrain the spatial and temporal consistency. In addition, we build a motion dataset that contains stable and unstable motion pairs for the training. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art online methods both qualitatively and quantitatively and achieves comparable performance to offline methods. Our code and dataset are available at https://github.com/liuzhen03/NNDVS