Diffusion Classifiers Understand Compositionality, but Conditions Apply

TL;DR

This paper provides a comprehensive analysis of diffusion classifiers' ability to understand compositionality across various models and datasets, revealing conditions that influence their performance and introducing a new diagnostic benchmark.

Contribution

It offers an extensive evaluation of diffusion classifiers on multiple datasets and tasks, introduces the Self-Bench diagnostic benchmark, and analyzes factors like domain effects and timestep sensitivity.

Findings

Diffusion classifiers understand compositionality under certain conditions.

Target dataset domain significantly affects classifier performance.

Timestep weighting impacts sensitivity to domain gaps, especially in SD3-m.

Abstract

Understanding visual scenes is fundamental to human intelligence. While discriminative models have significantly advanced computer vision, they often struggle with compositional understanding. In contrast, recent generative text-to-image diffusion models excel at synthesizing complex scenes, suggesting inherent compositional capabilities. Building on this, zero-shot diffusion classifiers have been proposed to repurpose diffusion models for discriminative tasks. While prior work offered promising results in discriminative compositional scenarios, these results remain preliminary due to a small number of benchmarks and a relatively shallow analysis of conditions under which the models succeed. To address this, we present a comprehensive study of the discriminative capabilities of diffusion classifiers on a wide range of compositional tasks. Specifically, our study covers three diffusion models (SD 1.5, 2.0, and, for the first time, 3-m) spanning 10 datasets and over 30 tasks. Further, we shed light on the role that target dataset domains play in respective performance; to isolate the domain effects, we introduce a new diagnostic benchmark \textsc{Self-Bench} comprised of images created by diffusion models themselves. Finally, we explore the importance of timestep weighting and uncover a relationship between domain gap and timestep sensitivity, particularly for SD3-m. To sum up, diffusion classifiers understand compositionality, but conditions apply! Code and dataset are available at https://github.com/eugene6923/Diffusion-Classifiers-Compositionality.