Rethinking Diffusion Posterior Sampling: From Conditional Score Estimator to Maximizing a Posterior

TL;DR

This paper reinterprets Diffusion Posterior Sampling (DPS) as a maximum a posteriori (MAP) approach rather than a conditional score estimator, and proposes improvements that significantly enhance its performance on image generation tasks.

Contribution

The paper demonstrates that DPS aligns more with MAP than conditional score estimation and introduces explicit posterior maximization and a lightweight score estimator to improve results.

Findings

DPS's conditional score diverges from the true score and is inferior to the unconditional score.

The conditional score estimate from DPS significantly deviates from zero, invalidating it as a proper score.

Proposed methods improve DPS's sample quality and diversity substantially.

Abstract

Recent advancements in diffusion models have been leveraged to address inverse problems without additional training, and Diffusion Posterior Sampling (DPS) (Chung et al., 2022a) is among the most popular approaches. Previous analyses suggest that DPS accomplishes posterior sampling by approximating the conditional score. While in this paper, we demonstrate that the conditional score approximation employed by DPS is not as effective as previously assumed, but rather aligns more closely with the principle of maximizing a posterior (MAP). This assertion is substantiated through an examination of DPS on 512x512 ImageNet images, revealing that: 1) DPS's conditional score estimation significantly diverges from the score of a well-trained conditional diffusion model and is even inferior to the unconditional score; 2) The mean of DPS's conditional score estimation deviates significantly from zero, rendering it an invalid score estimation; 3) DPS generates high-quality samples with significantly lower diversity. In light of the above findings, we posit that DPS more closely resembles MAP than a conditional score estimator, and accordingly propose the following enhancements to DPS: 1) we explicitly maximize the posterior through multi-step gradient ascent and projection; 2) we utilize a light-weighted conditional score estimator trained with only 100 images and 8 GPU hours. Extensive experimental results indicate that these proposed improvements significantly enhance DPS's performance. The source code for these improvements is provided in https://github.com/tongdaxu/Rethinking-Diffusion-Posterior-Sampling-From-Conditional-Score-Estimator-to-Maximizing-a-Posterior.