GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering

TL;DR

GaVS introduces a 3D-grounded video stabilization method that achieves temporally-consistent local reconstructions and rendering, reducing distortions and cropping issues, and improving overall stabilization quality.

Contribution

The paper presents a novel 3D-grounded approach reformulating video stabilization as local reconstruction and rendering, with test-time finetuning for temporal consistency and scene extrapolation to avoid cropping.

Findings

Outperforms state-of-the-art methods in geometry consistency.

Produces higher quality stabilized videos with less distortion.

Validated by user study showing improved user experience.

Abstract

Video stabilization is pivotal for video processing, as it removes unwanted shakiness while preserving the original user motion intent. Existing approaches, depending on the domain they operate, suffer from several issues (e.g. geometric distortions, excessive cropping, poor generalization) that degrade the user experience. To address these issues, we introduce \textbf{GaVS}, a novel 3D-grounded approach that reformulates video stabilization as a temporally-consistent `local reconstruction and rendering' paradigm. Given 3D camera poses, we augment a reconstruction model to predict Gaussian Splatting primitives, and finetune it at test-time, with multi-view dynamics-aware photometric supervision and cross-frame regularization, to produce temporally-consistent local reconstructions. The model are then used to render each stabilized frame. We utilize a scene extrapolation module to avoid frame cropping. Our method is evaluated on a repurposed dataset, instilled with 3D-grounded information, covering samples with diverse camera motions and scene dynamics. Quantitatively, our method is competitive with or superior to state-of-the-art 2D and 2.5D approaches in terms of conventional task metrics and new geometry consistency. Qualitatively, our method produces noticeably better results compared to alternatives, validated by the user study.