Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions

TL;DR

This paper introduces a synthetic data-based video stabilization method robust to adverse weather conditions, utilizing a novel rendering engine and a new dataset, outperforming existing approaches without needing real training data.

Contribution

The paper presents a synthetic-aware training approach for video stabilization that does not require real data, along with a new rendering engine and dataset for evaluation.

Findings

Our method outperforms five state-of-the-art algorithms across various adverse weather conditions.

Training solely on synthetic data yields models that generalize well to real-world videos.

The proposed approach achieves higher stability, lower distortion, and better cropping ratios in challenging conditions.

Abstract

Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our specially generated synthetic data for training an affine transformation matrix estimator avoiding the feature extraction issues faced by current methods. Additionally, since no video stabilization datasets under adverse conditions are available, we propose the novel VSAC105Real dataset for evaluation. We compare our method to five state-of-the-art video stabilization algorithms using two benchmarks. Our results show that current approaches perform poorly in at least one weather condition, and that, even training in a small dataset with synthetic data only, we achieve the best performance in terms of stability average score, distortion score, success rate, and average cropping ratio when considering all weather conditions. Hence, our video stabilization model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.