Patch-Based Diffusion Models Beat Whole-Image Models for Mismatched Distribution Inverse Problems

TL;DR

This paper introduces a patch-based diffusion model approach for inverse problems that outperforms whole-image models in mismatched distribution scenarios, especially with limited or single test samples, reducing artifacts and hallucinations.

Contribution

The authors propose a novel patch-based diffusion prior that learns from image patches, enabling high-quality reconstructions under distribution mismatch with minimal training data.

Findings

Patch-based models outperform whole-image models in OOD settings.

Patch approach reduces artifacts and hallucinations.

Method competes with models trained on large in-distribution datasets.

Abstract

Diffusion models have achieved excellent success in solving inverse problems due to their ability to learn strong image priors, but existing approaches require a large training dataset of images that should come from the same distribution as the test dataset. When the training and test distributions are mismatched, artifacts and hallucinations can occur in reconstructed images due to the incorrect priors. In this work, we systematically study out of distribution (OOD) problems where a known training distribution is first provided. We first study the setting where only a single measurement obtained from the unknown test distribution is available. Next we study the setting where a very small sample of data belonging to the test distribution is available, and our goal is still to reconstruct an image from a measurement that came from the test distribution. In both settings, we use a patch-based diffusion prior that learns the image distribution solely from patches. Furthermore, in the first setting, we include a self-supervised loss that helps the network output maintain consistency with the measurement. Extensive experiments show that in both settings, the patch-based method can obtain high quality image reconstructions that can outperform whole-image models and can compete with methods that have access to large in-distribution training datasets. Furthermore, we show how whole-image models are prone to memorization and overfitting, leading to artifacts in the reconstructions, while a patch-based model can resolve these issues.