Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

TL;DR

This paper investigates the ability of diffusion models to learn hidden inter-feature rules in images, revealing their limitations in capturing fine-grained relationships and analyzing the reasons behind these failures.

Contribution

It provides empirical evidence of diffusion models' struggles with fine-grained rules, introduces synthetic tasks for assessment, and offers theoretical insights into the limitations of denoising score matching.

Findings

Diffusion models fail to learn fine-grained inter-feature rules.

Incorporating classifier guidance offers limited improvements.

Theoretical analysis shows DSM objectives cause constant errors in rule learning.

Abstract

Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e.g., the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.