Latent Temporal Discrepancy as Motion Prior: A Loss-Weighting Strategy for Dynamic Fidelity in T2V

TL;DR

This paper introduces Latent Temporal Discrepancy (LTD), a motion prior that adaptively weights loss based on frame-to-frame latent variations, significantly improving dynamic fidelity in video generation models.

Contribution

The paper proposes LTD as a novel motion prior that guides loss weighting in diffusion models, enhancing their ability to generate high-quality dynamic videos.

Findings

Outperforms baselines by 3.31% on VBench

Outperforms baselines by 3.58% on VMBench

Improves motion quality in dynamic video generation

Abstract

Video generation models have achieved notable progress in static scenarios, yet their performance in motion video generation remains limited, with quality degrading under drastic dynamic changes. This is due to noise disrupting temporal coherence and increasing the difficulty of learning dynamic regions. {Unfortunately, existing diffusion models rely on static loss for all scenarios, constraining their ability to capture complex dynamics.} To address this issue, we introduce Latent Temporal Discrepancy (LTD) as a motion prior to guide loss weighting. LTD measures frame-to-frame variation in the latent space, assigning larger penalties to regions with higher discrepancy while maintaining regular optimization for stable regions. This motion-aware strategy stabilizes training and enables the model to better reconstruct high-frequency dynamics. Extensive experiments on the general benchmark VBench and the motion-focused VMBench show consistent gains, with our method outperforming strong baselines by 3.31% on VBench and 3.58% on VMBench, achieving significant improvements in motion quality.