DiffDoctor: Diagnosing Image Diffusion Models Before Treating

TL;DR

DiffDoctor introduces a two-stage approach with a robust artifact detector and pixel-level feedback to improve image diffusion models by reducing artifacts and enhancing image quality.

Contribution

The paper presents a novel two-stage pipeline with a large dataset and human-in-the-loop annotation for artifact detection and model optimization.

Findings

Effective artifact detection on over 1M images

Improved diffusion model quality with pixel-level feedback

Robustness of the proposed artifact detector

Abstract

In spite of recent progress, image diffusion models still produce artifacts. A common solution is to leverage the feedback provided by quality assessment systems or human annotators to optimize the model, where images are generally rated in their entirety. In this work, we believe problem-solving starts with identification, yielding the request that the model should be aware of not just the presence of defects in an image, but their specific locations. Motivated by this, we propose DiffDoctor, a two-stage pipeline to assist image diffusion models in generating fewer artifacts. Concretely, the first stage targets developing a robust artifact detector, for which we collect a dataset of over 1M flawed synthesized images and set up an efficient human-in-the-loop annotation process, incorporating a carefully designed class-balance strategy. The learned artifact detector is then involved in the second stage to optimize the diffusion model by providing pixel-level feedback. Extensive experiments on text-to-image diffusion models demonstrate the effectiveness of our artifact detector as well as the soundness of our diagnose-then-treat design.