A Simple and Efficient Baseline for Zero-Shot Generative Classification

TL;DR

This paper introduces a simple, fast, and accurate zero-shot diffusion classifier using pretrained models, achieving over 10 points better accuracy and 30,000 times faster speed than previous methods on ImageNet.

Contribution

It proposes the first zero-shot diffusion classifier that combines high accuracy with practical speed, leveraging pretrained text-to-image diffusion models and DINOv2.

Findings

Surpasses previous zero-shot diffusion classifiers by over 10 points on ImageNet.

Accelerates classification speed by more than 30,000 times.

Achieves competitive zero-shot performance across various datasets.

Abstract

Large diffusion models have become mainstream generative models in both academic studies and industrial AIGC applications. Recently, a number of works further explored how to employ the power of large diffusion models as zero-shot classifiers. While recent zero-shot diffusion-based classifiers have made performance advancement on benchmark datasets, they still suffered badly from extremely slow classification speed (e.g., ~1000 seconds per classifying single image on ImageNet). The extremely slow classification speed strongly prohibits existing zero-shot diffusion-based classifiers from practical applications. In this paper, we propose an embarrassingly simple and efficient zero-shot Gaussian Diffusion Classifiers (GDC) via pretrained text-to-image diffusion models and DINOv2. The proposed GDC can not only significantly surpass previous zero-shot diffusion-based classifiers by over 10 points (61.40% - 71.44%) on ImageNet, but also accelerate more than 30000 times (1000 - 0.03 seconds) classifying a single image on ImageNet. Additionally, it provides probability interpretation of the results. Our extensive experiments further demonstrate that GDC can achieve highly competitive zero-shot classification performance over various datasets and can promisingly self-improve with stronger diffusion models. To the best of our knowledge, the proposed GDC is the first zero-shot diffusionbased classifier that exhibits both competitive accuracy and practical efficiency.