Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization

TL;DR

This paper introduces a rapid, adaptive method for full-frame video stabilization that enhances stability and visual quality by focusing on high-jerk segments, outperforming existing state-of-the-art approaches.

Contribution

It presents a novel test-time adaptation strategy utilizing low-level cues and jerk localization to improve pixel-level video stabilization in diverse real-world scenarios.

Findings

Significant performance improvements over SOTA methods.

Effective stabilization with fewer adaptation steps.

Enhanced visual quality and stability across datasets.

Abstract

Video stabilization remains a fundamental problem in computer vision, particularly pixel-level synthesis solutions for video stabilization, which synthesize full-frame outputs, add to the complexity of this task. These methods aim to enhance stability while synthesizing full-frame videos, but the inherent diversity in motion profiles and visual content present in each video sequence makes robust generalization with fixed parameters difficult. To address this, we present a novel method that improves pixel-level synthesis video stabilization methods by rapidly adapting models to each input video at test time. The proposed approach takes advantage of low-level visual cues available during inference to improve both the stability and visual quality of the output. Notably, the proposed rapid adaptation achieves significant performance gains even with a single adaptation pass. We further propose a jerk localization module and a targeted adaptation strategy, which focuses the adaptation on high-jerk segments for maximizing stability with fewer adaptation steps. The proposed methodology enables modern stabilizers to overcome the longstanding SOTA approaches while maintaining the full frame nature of the modern methods, while offering users with control mechanisms akin to classical approaches. Extensive experiments on diverse real-world datasets demonstrate the versatility of the proposed method. Our approach consistently improves the performance of various full-frame synthesis models in both qualitative and quantitative terms, including results on downstream applications.