Repulsive Latent Score Distillation for Solving Inverse Problems

TL;DR

This paper introduces a novel variational framework with a repulsion mechanism to improve diversity and address latent space ambiguity in score distillation sampling for inverse problems, especially in high-dimensional data.

Contribution

It proposes a multimodal variational approximation with a repulsion mechanism and an augmented distribution to enhance diversity and disentangle latent space in SDS.

Findings

Improved solution diversity in inverse problems.

Enhanced performance on high-resolution image tasks.

Effective mitigation of mode collapse and latent ambiguity.

Abstract

Score Distillation Sampling (SDS) has been pivotal for leveraging pre-trained diffusion models in downstream tasks such as inverse problems, but it faces two major challenges: $(i)$ mode collapse and $(ii)$ latent space inversion, which become more pronounced in high-dimensional data. To address mode collapse, we introduce a novel variational framework for posterior sampling. Utilizing the Wasserstein gradient flow interpretation of SDS, we propose a multimodal variational approximation with a repulsion mechanism that promotes diversity among particles by penalizing pairwise kernel-based similarity. This repulsion acts as a simple regularizer, encouraging a more diverse set of solutions. To mitigate latent space ambiguity, we extend this framework with an augmented variational distribution that disentangles the latent and data. This repulsive augmented formulation balances computational efficiency, quality, and diversity. Extensive experiments on linear and nonlinear inverse tasks with high-resolution images ($512 \times 512$) using pre-trained Stable Diffusion models demonstrate the effectiveness of our approach.