EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis

TL;DR

EruDiff introduces a method to improve text-to-image diffusion models by refactoring their knowledge structures, enabling better handling of implicit prompts requiring deep world knowledge, and demonstrates significant performance gains.

Contribution

The paper proposes DK-DM and NO-RL strategies to align knowledge distributions and correct biases in diffusion models, enhancing their ability to process implicit prompts.

Findings

Significant performance improvements on Science-T2I and WISE benchmarks.

Enhanced handling of implicit prompts requiring deep world knowledge.

Effective knowledge refactoring in diffusion models demonstrated.

Abstract

Text-to-image diffusion models have achieved remarkable fidelity in synthesizing images from explicit text prompts, yet exhibit a critical deficiency in processing implicit prompts that require deep-level world knowledge, ranging from natural sciences to cultural commonsense, resulting in counter-factual synthesis. This paper traces the root of this limitation to a fundamental dislocation of the underlying knowledge structures, manifesting as a chaotic organization of implicit prompts compared to their explicit counterparts. In this paper, we propose EruDiff, which aims to refactor the knowledge within diffusion models. Specifically, we develop the Diffusion Knowledge Distribution Matching (DK-DM) to register the knowledge distribution of intractable implicit prompts with that of well-defined explicit anchors. Furthermore, to rectify the inherent biases in explicit prompt rendering, we employ the Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction. Rigorous empirical evaluations demonstrate that our method significantly enhances the performance of leading diffusion models, including FLUX and Qwen-Image, across both the scientific knowledge benchmark (i.e., Science-T2I) and the world knowledge benchmark (i.e., WISE), underscoring the effectiveness and generalizability. Our code is available at https://github.com/xiefan-guo/erudiff.