Learning Truncated Causal History Model for Video Restoration

TL;DR

TURTLE introduces a novel truncated causal history model for efficient, high-quality video restoration by summarizing historical frame information with a similarity-based retrieval mechanism, achieving state-of-the-art results.

Contribution

The paper presents TURTLE, a new method that models truncated causal history for video restoration, improving efficiency and performance over traditional methods.

Findings

Achieves state-of-the-art results on multiple video restoration benchmarks.

Reduces computational cost compared to existing methods.

Effectively models motion and alignment through a similarity-based retrieval mechanism.

Abstract

One key challenge to video restoration is to model the transition dynamics of video frames governed by motion. In this work, we propose TURTLE to learn the truncated causal history model for efficient and high-performing video restoration. Unlike traditional methods that process a range of contextual frames in parallel, TURTLE enhances efficiency by storing and summarizing a truncated history of the input frame latent representation into an evolving historical state. This is achieved through a sophisticated similarity-based retrieval mechanism that implicitly accounts for inter-frame motion and alignment. The causal design in TURTLE enables recurrence in inference through state-memorized historical features while allowing parallel training by sampling truncated video clips. We report new state-of-the-art results on a multitude of video restoration benchmark tasks, including video desnowing, nighttime video deraining, video raindrops and rain streak removal, video super-resolution, real-world and synthetic video deblurring, and blind video denoising while reducing the computational cost compared to existing best contextual methods on all these tasks.