Improving the Euclidean Diffusion Generation of Manifold Data by Mitigating Score Function Singularity

TL;DR

This paper addresses the challenge of score function singularity in Euclidean diffusion models for manifold data, proposing two novel methods to improve sampling accuracy and demonstrating their effectiveness on complex manifold distributions.

Contribution

The paper introduces Niso-DM and Tango-DM, two new techniques to mitigate score function singularity in Euclidean diffusion models for manifold data, enhancing sampling precision.

Findings

Niso-DM reduces scale discrepancies using non-isotropic noise.

Tango-DM trains only the tangential score component.

Methods outperform existing approaches on complex manifold distributions.

Abstract

Euclidean diffusion models have achieved remarkable success in generative modeling across diverse domains, and they have been extended to manifold cases in recent advances. Instead of explicitly utilizing the structure of special manifolds as studied in previous works, in this paper we investigate direct sampling of the Euclidean diffusion models for general manifold-structured data. We reveal the multiscale singularity of the score function in the ambient space, which hinders the accuracy of diffusion-generated samples. We then present an elaborate theoretical analysis of the singularity structure of the score function by decomposing it along the tangential and normal directions of the manifold. To mitigate the singularity and improve the sampling accuracy, we propose two novel methods: (1) Niso-DM, which reduces the scale discrepancies in the score function by utilizing a non-isotropic noise, and (2) Tango-DM, which trains only the tangential component of the score function using a tangential-only loss function. Numerical experiments demonstrate that our methods achieve superior performance on distributions over various manifolds with complex geometries.