Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentions

TL;DR

This paper explores the impact of CLIP image embeddings on video generation within the SVD framework, proposing a new efficient method that reduces computational load by replacing cross-attention with a linear layer, without sacrificing quality.

Contribution

It introduces VCUT, a training-free, efficient approach that eliminates the need for continuous cross-attention during inference, significantly reducing computational costs and latency.

Findings

CLIP embeddings are crucial for aesthetic quality but not for subject/background consistency.

Replacing cross-attention with a linear layer maintains quality while improving efficiency.

VCUT reduces MACs by up to 322T and decreases model parameters by 50M, with a 20% latency reduction.

Abstract

This paper investigates the role of CLIP image embeddings within the Stable Video Diffusion (SVD) framework, focusing on their impact on video generation quality and computational efficiency. Our findings indicate that CLIP embeddings, while crucial for aesthetic quality, do not significantly contribute towards the subject and background consistency of video outputs. Moreover, the computationally expensive cross-attention mechanism can be effectively replaced by a simpler linear layer. This layer is computed only once at the first diffusion inference step, and its output is then cached and reused throughout the inference process, thereby enhancing efficiency while maintaining high-quality outputs. Building on these insights, we introduce the VCUT, a training-free approach optimized for efficiency within the SVD architecture. VCUT eliminates temporal cross-attention and replaces spatial cross-attention with a one-time computed linear layer, significantly reducing computational load. The implementation of VCUT leads to a reduction of up to 322T Multiple-Accumulate Operations (MACs) per video and a decrease in model parameters by up to 50M, achieving a 20% reduction in latency compared to the baseline. Our approach demonstrates that conditioning during the Semantic Binding stage is sufficient, eliminating the need for continuous computation across all inference steps and setting a new standard for efficient video generation.