Grounding Degradations in Natural Language for All-In-One Video Restoration

TL;DR

This paper introduces an all-in-one video restoration framework that uses foundation models to interpret degradation-aware semantic context from natural language, enabling flexible, interpretable, and degradation-agnostic video enhancement.

Contribution

It proposes a novel framework that grounds degradation knowledge in natural language without requiring degradation info during training or testing, and introduces standardized benchmarks for multi-degradation video restoration.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Demonstrates flexibility without degradation knowledge during inference.

Provides new datasets simulating weather-related degradations.

Abstract

In this work, we propose an all-in-one video restoration framework that grounds degradation-aware semantic context of video frames in natural language via foundation models, offering interpretable and flexible guidance. Unlike prior art, our method assumes no degradation knowledge in train or test time and learns an approximation to the grounded knowledge such that the foundation model can be safely disentangled during inference adding no extra cost. Further, we call for standardization of benchmarks in all-in-one video restoration, and propose two benchmarks in multi-degradation setting, three-task (3D) and four-task (4D), and two time-varying composite degradation benchmarks; one of the latter being our proposed dataset with varying snow intensity, simulating how weather degradations affect videos naturally. We compare our method with prior works and report state-of-the-art performance on all benchmarks.