Transform Trained Transformer: Accelerating Naive 4K Video Generation Over 10$\times$

TL;DR

This paper introduces T3, a retrofit strategy for pretrained Transformers that significantly accelerates 4K video generation by optimizing attention mechanisms without changing core architecture.

Contribution

T3 provides a novel, architecture-preserving method to reduce computational costs of full-attention models for high-resolution video generation.

Findings

Over 10× acceleration in 4K video generation

Performance improvements in VQA and VTC metrics

Effective attention pattern transformation with modest compute

Abstract

Native 4K (2160$\times$3840) video generation remains a critical challenge due to the quadratic computational explosion of full-attention as spatiotemporal resolution increases, making it difficult for models to strike a balance between efficiency and quality. This paper proposes a novel Transformer retrofit strategy termed $\textbf{T3}$ ($\textbf{T}$ransform $\textbf{T}$rained $\textbf{T}$ransformer) that, without altering the core architecture of full-attention pretrained models, significantly reduces compute requirements by optimizing their forward logic. Specifically, $\textbf{T3-Video}$ introduces a multi-scale weight-sharing window attention mechanism and, via hierarchical blocking together with an axis-preserving full-attention design, can effect an "attention pattern" transformation of a pretrained model using only modest compute and data. Results on 4K-VBench show that $\textbf{T3-Video}$ substantially outperforms existing approaches: while delivering performance improvements (+4.29$\uparrow$ VQA and +0.08$\uparrow$ VTC), it accelerates native 4K video generation by more than 10$\times$. Project page at https://zhangzjn.github.io/projects/T3-Video