T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates

TL;DR

T-GVC introduces a novel generative video coding framework that combines semantic-aware sparse motion sampling with trajectory-aligned loss constraints, enabling ultra-low bitrate video compression with realistic motion and high semantic fidelity.

Contribution

It proposes a trajectory-guided approach that reduces bitrate while maintaining motion quality, integrating motion tracking with semantic understanding in a training-free guidance mechanism.

Findings

Outperforms traditional and neural codecs at ultra-low bitrates

Achieves more precise motion control than text-guided methods

Preserves critical semantic and motion details in compressed videos

Abstract

Recent advances in video generation techniques have given rise to an emerging paradigm of generative video coding for Ultra-Low Bitrate (ULB) scenarios by leveraging powerful generative priors. However, most existing methods are limited by domain specificity (e.g., facial or human videos) or excessive dependence on high-level text guidance, which tend to inadequately capture fine-grained motion details, leading to unrealistic or incoherent reconstructions. To address these challenges, we propose Trajectory-Guided Generative Video Coding (dubbed T-GVC), a novel framework that bridges low-level motion tracking with high-level semantic understanding. T-GVC features a semantic-aware sparse motion sampling pipeline that extracts pixel-wise motion as sparse trajectory points based on their semantic importance, significantly reducing the bitrate while preserving critical temporal semantic information. In addition, by integrating trajectory-aligned loss constraints into diffusion processes, we introduce a training-free guidance mechanism in latent space to ensure physically plausible motion patterns without sacrificing the inherent capabilities of generative models. Experimental results demonstrate that T-GVC outperforms both traditional and neural video codecs under ULB conditions. Furthermore, additional experiments confirm that our framework achieves more precise motion control than existing text-guided methods, paving the way for a novel direction of generative video coding guided by geometric motion modeling.