Reviving ConvNeXt for Efficient Convolutional Diffusion Models

TL;DR

This paper introduces FCDM, a convolutional diffusion model based on ConvNeXt, demonstrating that modern convolutional architectures can achieve competitive performance with significantly reduced computational cost and training resources.

Contribution

The paper revives ConvNeXt for diffusion modeling, showing it can outperform transformer-based models in efficiency and training simplicity for generative tasks.

Findings

FCDM-XL achieves comparable performance with 50% FLOPs of DiT-XL/2.

FCDM-XL requires 7-7.5 times fewer training steps at high resolutions.

FCDM-XL can be trained on a 4-GPU system, highlighting its efficiency.

Abstract

Recent diffusion models increasingly favor Transformer backbones, motivated by the remarkable scalability of fully attentional architectures. Yet the locality bias, parameter efficiency, and hardware friendliness--the attributes that established ConvNets as the efficient vision backbone--have seen limited exploration in modern generative modeling. Here we introduce the fully convolutional diffusion model (FCDM), a model having a backbone similar to ConvNeXt, but designed for conditional diffusion modeling. We find that using only 50% of the FLOPs of DiT-XL/2, FCDM-XL achieves competitive performance with 7$\times$ and 7.5$\times$ fewer training steps at 256$\times$256 and 512$\times$512 resolutions, respectively. Remarkably, FCDM-XL can be trained on a 4-GPU system, highlighting the exceptional training efficiency of our architecture. Our results demonstrate that modern convolutional designs provide a competitive and highly efficient alternative for scaling diffusion models, reviving ConvNeXt as a simple yet powerful building block for efficient generative modeling.