Task Agnostic Restoration of Natural Video Dynamics

TL;DR

This paper introduces a novel framework that infers and utilizes consistent motion dynamics from inconsistent videos to improve temporal consistency in video restoration tasks without needing raw videos at test time.

Contribution

It presents a general, learning-based approach that enhances temporal consistency in video processing by modeling motion dynamics, outperforming existing methods on benchmark datasets.

Findings

Achieves state-of-the-art results on DAVIS and videvo.net datasets.

Effectively reduces temporal flicker in processed videos.

Preserves perceptual quality across frames.

Abstract

In many video restoration/translation tasks, image processing operations are na\"ively extended to the video domain by processing each frame independently, disregarding the temporal connection of the video frames. This disregard for the temporal connection often leads to severe temporal inconsistencies. State-Of-The-Art (SOTA) techniques that address these inconsistencies rely on the availability of unprocessed videos to implicitly siphon and utilize consistent video dynamics to restore the temporal consistency of frame-wise processed videos which often jeopardizes the translation effect. We propose a general framework for this task that learns to infer and utilize consistent motion dynamics from inconsistent videos to mitigate the temporal flicker while preserving the perceptual quality for both the temporally neighboring and relatively distant frames without requiring the raw videos at test time. The proposed framework produces SOTA results on two benchmark datasets, DAVIS and videvo.net, processed by numerous image processing applications. The code and the trained models are available at \url{https://github.com/MKashifAli/TARONVD}.