Refining Visual Artifacts in Diffusion Models via Explainable AI-based Flaw Activation Maps

TL;DR

This paper introduces a self-refining diffusion framework that uses explainable AI to detect and amplify flaws in generated images, significantly improving image quality across multiple tasks and models.

Contribution

It presents a novel flaw detection and enhancement method using flaw activation maps, advancing diffusion model refinement with explainable AI techniques.

Findings

Up to 27.3% improvement in FID scores

Effective across diverse datasets and tasks

Robust performance on various diffusion models

Abstract

Diffusion models have achieved remarkable success in image synthesis. However, addressing artifacts and unrealistic regions remains a critical challenge. We propose self-refining diffusion, a novel framework that enhances image generation quality by detecting these flaws. The framework employs an explainable artificial intelligence (XAI)-based flaw highlighter to produce flaw activation maps (FAMs) that identify artifacts and unrealistic regions. These FAMs improve reconstruction quality by amplifying noise in flawed regions during the forward process and by focusing on these regions during the reverse process. The proposed approach achieves up to a 27.3% improvement in Fr\'echet inception distance across various diffusion-based models, demonstrating consistently strong performance on diverse datasets. It also shows robust effectiveness across different tasks, including image generation, text-to-image generation, and inpainting. These results demonstrate that explainable AI techniques can extend beyond interpretability to actively contribute to image refinement. The proposed framework offers a versatile and effective approach applicable to various diffusion models and tasks, significantly advancing the field of image synthesis.