Towards Precise Scaling Laws for Video Diffusion Transformers

TL;DR

This paper systematically analyzes and confirms the existence of scaling laws for video diffusion transformers, proposing a new law to optimize hyperparameters and improve performance-cost trade-offs.

Contribution

It introduces a novel scaling law for video diffusion transformers that predicts optimal hyperparameters and enhances performance within compute constraints.

Findings

Confirmed the presence of scaling laws in video diffusion models

Discovered sensitivity to learning rate and batch size

Reduced inference costs by 40.1% under the same compute budget

Abstract

Achieving optimal performance of video diffusion transformers within given data and compute budget is crucial due to their high training costs. This necessitates precisely determining the optimal model size and training hyperparameters before large-scale training. While scaling laws are employed in language models to predict performance, their existence and accurate derivation in visual generation models remain underexplored. In this paper, we systematically analyze scaling laws for video diffusion transformers and confirm their presence. Moreover, we discover that, unlike language models, video diffusion models are more sensitive to learning rate and batch size, two hyperparameters often not precisely modeled. To address this, we propose a new scaling law that predicts optimal hyperparameters for any model size and compute budget. Under these optimal settings, we achieve comparable performance and reduce inference costs by 40.1% compared to conventional scaling methods, within a compute budget of 1e10 TFlops. Furthermore, we establish a more generalized and precise relationship among validation loss, any model size, and compute budget. This enables performance prediction for non-optimal model sizes, which may also be appealed under practical inference cost constraints, achieving a better trade-off.