Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements

TL;DR

This paper introduces a novel internal learning framework for robust video enhancement tasks that learns directly from the corrupted test video without external data, utilizing a spatial pyramid loss to improve robustness and achieve state-of-the-art results.

Contribution

It proposes a new internal learning approach that optimizes neural modules directly on test videos, incorporating a spatial pyramid loss for enhanced robustness in various low-level vision tasks.

Findings

Achieves state-of-the-art results in denoising, object removal, and frame interpolation.

Demonstrates robustness to unstructured noise and input degradation.

Validated on standard datasets like DAVIS, UCF-101, and VIMEO90K-T.

Abstract

In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights of neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce a novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in a video across the different scales of the video. This loss enhances robustness to unstructured noise in both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the-art results on downstream tasks such as denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations on standard video datasets such as DAVIS, UCF-101, and VIMEO90K-T.