Prototype Clustered Diffusion Models for Versatile Inverse Problems

TL;DR

This paper introduces Prototype Clustered Diffusion Models that leverage restoration-based likelihoods and prototypes to improve inverse problem solving, offering adaptable, realistic, and versatile solutions across various applications.

Contribution

It proposes a novel probabilistic framework extending deterministic models to clustered processes with prototypes, enabling flexible, restoration-guided inverse problem solutions.

Findings

Effective in image dehazing, rain streak removal, and motion deblurring

Outperforms existing methods in sample quality and realism

Provides adaptable control over deterioration processes

Abstract

Diffusion models have made remarkable progress in solving various inverse problems, attributing to the generative modeling capability of the data manifold. Posterior sampling from the conditional score function enable the precious data consistency certified by the measurement-based likelihood term. However, most prevailing approaches confined to the deterministic deterioration process of the measurement model, regardless of capricious unpredictable disturbance in real-world sceneries. To address this obstacle, we show that the measurement-based likelihood can be renovated with restoration-based likelihood via the opposite probabilistic graphic direction, licencing the patronage of various off-the-shelf restoration models and extending the strictly deterministic deterioration process to adaptable clustered processes with the supposed prototype, in what we call restorer guidance. Particularly, assembled with versatile prototypes optionally, we can resolve inverse problems with bunch of choices for assorted sample quality and realize the proficient deterioration control with assured realistic. We show that our work can be formally analogous to the transition from classifier guidance to classifier-free guidance in the field of inverse problem solver. Experiments on multifarious inverse problems demonstrate the effectiveness of our method, including image dehazing, rain streak removal, and motion deblurring.