VS3R: Robust Full-frame Video Stabilization via Deep 3D Reconstruction

TL;DR

VS3R is a novel full-frame video stabilization framework that combines 3D reconstruction, semantic-guided rendering, and diffusion models to achieve robust, high-quality stabilization across diverse scenarios.

Contribution

It introduces a hybrid pipeline integrating 3D reconstruction with generative diffusion, enhancing robustness and visual quality in full-frame video stabilization.

Findings

Outperforms state-of-the-art methods in robustness and quality

Achieves high-fidelity stabilization across diverse camera models

Effectively restores disoccluded regions and corrects artifacts

Abstract

Video stabilization aims to mitigate camera shake but faces a fundamental trade-off between geometric robustness and full-frame consistency. While 2D methods suffer from aggressive cropping, 3D techniques are often undermined by fragile optimization pipelines that fail under extreme motions. To bridge this gap, we propose VS3R, a framework that synergizes feed-forward 3D reconstruction with generative video diffusion. Our pipeline jointly estimates camera parameters, depth, and masks to ensure all-scenario reliability, and introduces a Hybrid Stabilized Rendering module that fuses semantic and geometric cues for dynamic consistency. Finally, a Dual-Stream Video Diffusion Model restores disoccluded regions and rectifies artifacts by synergizing structural guidance with semantic anchors. Collectively, VS3R achieves high-fidelity, full-frame stabilization across diverse camera models and significantly outperforms state-of-the-art methods in robustness and visual quality.