Laplacian Score Sharpening for Mitigating Hallucination in Diffusion Models

TL;DR

This paper introduces a post-hoc Laplacian-based adjustment to the score function in diffusion models, effectively reducing hallucinations and mode interpolation during sampling across various data dimensions.

Contribution

It proposes a novel Laplacian sharpening method for diffusion models that mitigates hallucinations by adjusting the score function during inference, with an efficient approximation for high-dimensional data.

Findings

Significant reduction in hallucinated samples in toy and image datasets

Effective Laplacian approximation for high-dimensional data

Analysis of Laplacian's relation to score uncertainty

Abstract

Diffusion models, though successful, are known to suffer from hallucinations that create incoherent or unrealistic samples. Recent works have attributed this to the phenomenon of mode interpolation and score smoothening, but they lack a method to prevent their generation during sampling. In this paper, we propose a post-hoc adjustment to the score function during inference that leverages the Laplacian (or sharpness) of the score to reduce mode interpolation hallucination in unconditional diffusion models across 1D, 2D, and high-dimensional image data. We derive an efficient Laplacian approximation for higher dimensions using a finite-difference variant of the Hutchinson trace estimator. We show that this correction significantly reduces the rate of hallucinated samples across toy 1D/2D distributions and a high-dimensional image dataset. Furthermore, our analysis explores the relationship between the Laplacian and uncertainty in the score.