DeRA: Decoupled Representation Alignment for Video Tokenization

TL;DR

DeRA introduces a decoupled 1D video tokenizer that separately models appearance and motion, improving training efficiency and performance by aligning with pretrained models and addressing gradient conflicts.

Contribution

The paper proposes DeRA, a novel video tokenizer that decouples spatial and temporal representations and introduces SACP to handle gradient conflicts, achieving state-of-the-art results.

Findings

DeRA outperforms previous methods by 25% on UCF-101 rFVD.

DeRA achieves new SOTA in video generation and frame prediction.

Decoupled representation improves training efficiency and accuracy.

Abstract

This paper presents DeRA, a novel 1D video tokenizer that decouples the spatial-temporal representation learning in video tokenization to achieve better training efficiency and performance. Specifically, DeRA maintains a compact 1D latent space while factorizing video encoding into appearance and motion streams, which are aligned with pretrained vision foundation models to capture the spatial semantics and temporal dynamics in videos separately. To address the gradient conflicts introduced by the heterogeneous supervision, we further propose the Symmetric Alignment-Conflict Projection (SACP) module that proactively reformulates gradients by suppressing the components along conflicting directions. Extensive experiments demonstrate that DeRA outperforms LARP, the previous state-of-the-art video tokenizer by 25% on UCF-101 in terms of rFVD. Moreover, using DeRA for autoregressive video generation, we also achieve new state-of-the-art results on both UCF-101 class-conditional generation and K600 frame prediction.