DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability

TL;DR

DiffDis introduces a unified diffusion framework that jointly models image generation and cross-modal discrimination, enhancing both tasks by sharing architecture and leveraging diffusion-based training.

Contribution

The paper proposes DiffDis, a novel dual-stream diffusion model that unifies generative and discriminative cross-modal learning in a single architecture.

Findings

Outperforms single-task models in image synthesis quality (FID improvement).

Achieves higher accuracy in zero-shot image classification across multiple datasets.

Demonstrates effective cross-modal semantic alignment in a unified diffusion framework.

Abstract

Recently, large-scale diffusion models, e.g., Stable diffusion and DallE2, have shown remarkable results on image synthesis. On the other hand, large-scale cross-modal pre-trained models (e.g., CLIP, ALIGN, and FILIP) are competent for various downstream tasks by learning to align vision and language embeddings. In this paper, we explore the possibility of jointly modeling generation and discrimination. Specifically, we propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. DiffDis first formulates the image-text discriminative problem as a generative diffusion process of the text embedding from the text encoder conditioned on the image. Then, we propose a novel dual-stream network architecture, which fuses the noisy text embedding with the knowledge of latent images from different scales for image-text discriminative learning. Moreover, the generative and discriminative tasks can efficiently share the image-branch network structure in the multi-modality model. Benefiting from diffusion-based unified training, DiffDis achieves both better generation ability and cross-modal semantic alignment in one architecture. Experimental results show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks, e.g., 1.65% improvement on average accuracy of zero-shot classification over 12 datasets and 2.42 improvement on FID of zero-shot image synthesis.