EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing

TL;DR

EC-DIT introduces an adaptive Mixture-of-Experts approach for diffusion transformers, enabling efficient scaling to 97 billion parameters and improving text-to-image synthesis quality through heterogeneous compute allocation.

Contribution

The paper presents EC-DIT, a novel adaptive expert-choice routing method for diffusion transformers that enhances scalability and performance in text-to-image synthesis.

Findings

Achieved state-of-the-art GenEval score of 71.68%.

Enabled scaling of models up to 97 billion parameters.

Demonstrated improved training convergence and image quality.

Abstract

Diffusion transformers have been widely adopted for text-to-image synthesis. While scaling these models up to billions of parameters shows promise, the effectiveness of scaling beyond current sizes remains underexplored and challenging. By explicitly exploiting the computational heterogeneity of image generations, we develop a new family of Mixture-of-Experts (MoE) models (EC-DIT) for diffusion transformers with expert-choice routing. EC-DIT learns to adaptively optimize the compute allocated to understand the input texts and generate the respective image patches, enabling heterogeneous computation aligned with varying text-image complexities. This heterogeneity provides an efficient way of scaling EC-DIT up to 97 billion parameters and achieving significant improvements in training convergence, text-to-image alignment, and overall generation quality over dense models and conventional MoE models. Through extensive ablations, we show that EC-DIT demonstrates superior scalability and adaptive compute allocation by recognizing varying textual importance through end-to-end training. Notably, in text-to-image alignment evaluation, our largest models achieve a state-of-the-art GenEval score of 71.68% and still maintain competitive inference speed with intuitive interpretability.