Diffusion State-Guided Projected Gradient for Inverse Problems
Rayhan Zirvi, Bahareh Tolooshams, Anima Anandkumar
TL;DR
This paper introduces DiffStateGrad, a novel method that enhances diffusion model-based inverse problem solvers by projecting measurement guidance onto a low-rank subspace, improving robustness and image restoration quality.
Contribution
The paper proposes DiffStateGrad, a module that can be integrated into diffusion-based inverse solvers to better preserve the prior manifold and reduce artifacts, advancing the state-of-the-art.
Findings
Improves robustness to guidance step size and noise.
Enhances worst-case performance in inverse problems.
Outperforms existing methods in image restoration tasks.
Abstract
Recent advancements in diffusion models have been effective in learning data priors for solving inverse problems. They leverage diffusion sampling steps for inducing a data prior while using a measurement guidance gradient at each step to impose data consistency. For general inverse problems, approximations are needed when an unconditionally trained diffusion model is used since the measurement likelihood is intractable, leading to inaccurate posterior sampling. In other words, due to their approximations, these methods fail to preserve the generation process on the data manifold defined by the diffusion prior, leading to artifacts in applications such as image restoration. To enhance the performance and robustness of diffusion models in solving inverse problems, we propose Diffusion State-Guided Projected Gradient (DiffStateGrad), which projects the measurement gradient onto a subspace…
Peer Reviews
Decision·ICLR 2025 Poster
1. Well written 2. Easy to Understand 3. The idea of projecting the gradient to the manifold of intermediate noise is novel and making sense to me. This method supposes to suppress artifacts that arises with hard optimization.
1. In some cases where gradient computation may incur some additional burdens (for example when PSLD takes a lot of memory), this method may not be feasible. 2. There are some other works that try to project the restoration gradient onto the prior manifold (for example DreamClean [1], and MCG [2]). 3. The baselines with LDMs are sufficient in my view, but this paper could benefits more with pixel diffusion baselines such as DDNM [3], DDRM and so on. [1] Xiao, Jie, et al. "DreamClean: Restoring
* The paper is written very well. The work is contextualized well among related work and I enjoyed reading the paper. * DiffStateGrad is formulated as a module that greatly increases robustness of existing SOTA posterior sampling approaches (such as PSLD and DAPS) with respect to the choice of step siz and measurement noise. * The effectiveness of DiffStateGrad is demonstrated in diverse set of forward models such as box inpainting, random inpainting, Gaussian deblur, motion deblur, etc. for li
* See the questions below.
- The paper is well written and the methodology is clearly presented. The fact that the proposed algorithm can be plugged on top of existing methods is a significant feat. The theoretical justification for the method is loose but the explanations provided are intuitive. - The experiments are convincing are well thought and rather extensive.
- While the additional computational cost is marginal for latent diffusion models, isn't it prohibitive for pixel space diffusion models? Having to compute the SVD at each iteration is certainly a significant drawback. - The methodology is based on having at some point a sample on the manifold of "artifact free images" (this is loosely defined). It is unclear how this arises in practice. - There are some inconsistencies in the experiments which I believe are explained nowhere in the paper. Fir
Code & Models
Videos
Taxonomy
TopicsNumerical methods in inverse problems · Radiative Heat Transfer Studies · Advanced Mathematical Modeling in Engineering
MethodsDiffusion