Importance Weighted Score Matching for Diffusion Samplers with Enhanced Mode Coverage

TL;DR

This paper introduces Importance Weighted Score Matching, a novel training method for diffusion-based neural samplers that improves mode coverage by directly optimizing an objective akin to the forward KL divergence, especially in data-scarce scenarios.

Contribution

It proposes a principled importance weighted score matching approach that enhances mode coverage in neural samplers without relying on target samples, backed by theoretical analysis and empirical validation.

Findings

Outperforms existing neural samplers on complex multi-modal distributions.

Achieves state-of-the-art results on benchmarks with up to 120 modes.

Provides theoretical insights into bias and variance of the estimator.

Abstract

Training neural samplers directly from unnormalized densities without access to target distribution samples presents a significant challenge. A critical desideratum in these settings is achieving comprehensive mode coverage, ensuring the sampler captures the full diversity of the target distribution. However, prevailing methods often circumvent the lack of target data by optimizing reverse KL-based objectives. Such objectives inherently exhibit mode-seeking behavior, potentially leading to incomplete representation of the underlying distribution. While alternative approaches strive for better mode coverage, they typically rely on implicit mechanisms like heuristics or iterative refinement. In this work, we propose a principled approach for training diffusion-based samplers by directly targeting an objective analogous to the forward KL divergence, which is conceptually known to encourage mode coverage. We introduce \textit{Importance Weighted Score Matching}, a method that optimizes this desired mode-covering objective by re-weighting the score matching loss using tractable importance sampling estimates, thereby overcoming the absence of target distribution data. We also provide theoretical analysis of the bias and variance for our proposed Monte Carlo estimator and the practical loss function used in our method. Experiments on increasingly complex multi-modal distributions, including 2D Gaussian Mixture Models with up to 120 modes and challenging particle systems with inherent symmetries -- demonstrate that our approach consistently outperforms existing neural samplers across all distributional distance metrics, achieving state-of-the-art results on all benchmarks.