DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling

TL;DR

This paper introduces DiCo, a convolution-based diffusion model that replaces self-attention with efficient convolutional modules, achieving comparable or better performance with significantly reduced computational costs.

Contribution

The paper proposes a novel convolutional diffusion model, DiCo, with a channel attention mechanism to enhance feature diversity, replacing costly self-attention in diffusion transformers.

Findings

DiCo-XL achieves an FID of 2.05 on ImageNet at 256x256.

DiCo models are 2.7x to 3.1x faster than DiT-XL/2.

Purely convolutional DiCo performs well on text-to-image tasks.

Abstract

Diffusion Transformer (DiT), a promising diffusion model for visual generation, demonstrates impressive performance but incurs significant computational overhead. Intriguingly, analysis of pre-trained DiT models reveals that global self-attention is often redundant, predominantly capturing local patterns-highlighting the potential for more efficient alternatives. In this paper, we revisit convolution as an alternative building block for constructing efficient and expressive diffusion models. However, naively replacing self-attention with convolution typically results in degraded performance. Our investigations attribute this performance gap to the higher channel redundancy in ConvNets compared to Transformers. To resolve this, we introduce a compact channel attention mechanism that promotes the activation of more diverse channels, thereby enhancing feature diversity. This leads to Diffusion ConvNet (DiCo), a family of diffusion models built entirely from standard ConvNet modules, offering strong generative performance with significant efficiency gains. On class-conditional ImageNet generation benchmarks, DiCo-XL achieves an FID of 2.05 at 256x256 resolution and 2.53 at 512x512, with a 2.7x and 3.1x speedup over DiT-XL/2, respectively. Furthermore, experimental results on MS-COCO demonstrate that the purely convolutional DiCo exhibits strong potential for text-to-image generation. Code: https://github.com/shallowdream204/DiCo.