Corruption-Aware Training of Latent Video Diffusion Models for Robust Text-to-Video Generation

TL;DR

This paper introduces CAT-LVDM, a corruption-aware training framework for latent video diffusion models that enhances robustness against noisy conditioning through structured, data-aligned noise injection techniques.

Contribution

It proposes novel noise operators, BCNI and SACN, tailored for video data, improving robustness and performance over existing diffusion models and establishing theoretical robustness bounds.

Findings

BCNI reduces FVD by 31.9% on multiple datasets.

SACN improves UCF-101 accuracy by 12.3%.

Outperforms large diffusion baselines despite less training data.

Abstract

Latent Video Diffusion Models (LVDMs) have achieved state-of-the-art generative quality for image and video generation; however, they remain brittle under noisy conditioning, where small perturbations in text or multimodal embeddings can cascade over timesteps and cause semantic drift. Existing corruption strategies from image diffusion (Gaussian, Uniform) fail in video settings because static noise disrupts temporal fidelity. In this paper, we propose CAT-LVDM, a corruption-aware training framework with structured, data-aligned noise injection tailored for video diffusion. Our two operators, Batch-Centered Noise Injection (BCNI) and Spectrum-Aware Contextual Noise (SACN), align perturbations with batch semantics or spectral dynamics to preserve coherence. CAT-LVDM yields substantial gains: BCNI reduces FVD by 31.9 percent on WebVid-2M, MSR-VTT, and MSVD, while SACN improves UCF-101 by 12.3 percent, outperforming Gaussian, Uniform, and even large diffusion baselines like DEMO (2.3B) and Lavie (3B) despite training on 5x less data. Ablations confirm the unique value of low-rank, data-aligned noise, and theory establishes why these operators tighten robustness and generalization bounds. CAT-LVDM thus sets a new framework for robust video diffusion, and our experiments show that it can also be extended to autoregressive generation and multimodal video understanding LLMs. Code, models, and samples are available at https://github.com/chikap421/catlvdm