DiC: Rethinking Conv3x3 Designs in Diffusion Models

TL;DR

This paper introduces DiC, a convolution-based diffusion model that outperforms transformer-based models in speed and competitiveness by rethinking convolutional design and incorporating novel architectural and conditioning improvements.

Contribution

The paper proposes a purely convolutional diffusion architecture, DiC, with novel design enhancements like sparse skip connections and advanced conditioning, surpassing transformer-based models in performance and speed.

Findings

DiC outperforms diffusion transformers in quality.

DiC maintains faster inference speeds.

Architectural improvements significantly boost diffusion performance.

Abstract

Diffusion models have shown exceptional performance in visual generation tasks. Recently, these models have shifted from traditional U-Shaped CNN-Attention hybrid structures to fully transformer-based isotropic architectures. While these transformers exhibit strong scalability and performance, their reliance on complicated self-attention operation results in slow inference speeds. Contrary to these works, we rethink one of the simplest yet fastest module in deep learning, 3x3 Convolution, to construct a scaled-up purely convolutional diffusion model. We first discover that an Encoder-Decoder Hourglass design outperforms scalable isotropic architectures for Conv3x3, but still under-performing our expectation. Further improving the architecture, we introduce sparse skip connections to reduce redundancy and improve scalability. Based on the architecture, we introduce conditioning improvements including stage-specific embeddings, mid-block condition injection, and conditional gating. These improvements lead to our proposed Diffusion CNN (DiC), which serves as a swift yet competitive diffusion architecture baseline. Experiments on various scales and settings show that DiC surpasses existing diffusion transformers by considerable margins in terms of performance while keeping a good speed advantage. Project page: https://github.com/YuchuanTian/DiC