GenTron: Diffusion Transformers for Image and Video Generation
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie,, Animesh Sinha, Ping Luo, Tao Xiang, Juan-Manuel Perez-Rua

TL;DR
GenTron introduces Transformer-based diffusion models for image and video generation, achieving superior visual quality and text alignment, and extending the application of diffusion Transformers to visual content.
Contribution
This work pioneers the use of Transformer-based diffusion models for image and video generation, scaling models significantly and introducing novel motion-free guidance for videos.
Findings
GenTron outperforms SDXL in visual quality and text alignment.
Scaling from 900M to 3B parameters improves visual results.
GenTron excels in compositional generation benchmarks.
Abstract
In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexibility and scalability, the visual generative domain primarily utilizes CNN-based U-Net architectures, particularly in diffusion-based models. We introduce GenTron, a family of Generative models employing Transformer-based diffusion, to address this gap. Our initial step was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a process involving thorough empirical exploration of the conditioning mechanism. We then scale GenTron from approximately 900M to over 3B parameters, observing significant improvements in visual quality. Furthermore, we extend GenTron to text-to-video generation, incorporating novel motion-free guidance to enhance video quality. In human evaluations against SDXL,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis
MethodsAttention Is All You Need · Linear Layer · Residual Connection · Dropout · Softmax · Multi-Head Attention · Byte Pair Encoding · Adam · Absolute Position Encodings · Layer Normalization
