Blind Bitstream-corrupted Video Recovery via Metadata-guided Diffusion Model

TL;DR

This paper introduces a blind video recovery method using a metadata-guided diffusion model that does not require predefined masks, effectively restoring corrupted videos by leveraging intrinsic metadata and a novel mask prediction approach.

Contribution

The paper proposes a new blind video recovery framework that utilizes intrinsic metadata and a prior-driven mask predictor, eliminating the need for manual mask annotation.

Findings

The method effectively restores corrupted videos without predefined masks.

The dual-stream metadata encoder captures motion and frame type information.

Post-refinement improves boundary consistency between recovered and intact regions.

Abstract

Bitstream-corrupted video recovery aims to restore realistic content degraded during video storage or transmission. Existing methods typically assume that predefined masks of corrupted regions are available, but manually annotating these masks is labor-intensive and impractical in real-world scenarios. To address this limitation, we introduce a new blind video recovery setting that removes the reliance on predefined masks. This setting presents two major challenges: accurately identifying corrupted regions and recovering content from extensive and irregular degradations. We propose a Metadata-Guided Diffusion Model (M-GDM) to tackle these challenges. Specifically, intrinsic video metadata are leveraged as corruption indicators through a dual-stream metadata encoder that separately embeds motion vectors and frame types before fusing them into a unified representation. This representation interacts with corrupted latent features via cross-attention at each diffusion step. To preserve intact regions, we design a prior-driven mask predictor that generates pseudo masks using both metadata and diffusion priors, enabling the separation and recombination of intact and recovered regions through hard masking. To mitigate boundary artifacts caused by imperfect masks, a post-refinement module enhances consistency between intact and recovered regions. Extensive experiments demonstrate the effectiveness of our method and its superiority in blind video recovery. Code is available at: https://github.com/Shuyun-Wang/M-GDM.