Denoising Fisher Training For Neural Implicit Samplers

TL;DR

This paper introduces Denoising Fisher Training (DFT), a new method for training neural implicit samplers that improves sampling efficiency and mode coverage, with theoretical guarantees and strong empirical results in high-dimensional settings.

Contribution

The paper proposes DFT, a novel training approach that minimizes Fisher divergence with a tractable loss, providing theoretical guarantees and improved sampling performance.

Findings

DFT achieves high-quality sampling with fewer steps, comparable to MCMC.

Empirical validation across diverse benchmarks demonstrates superior efficiency.

High-dimensional EBM sampling shows over 100x efficiency gains.

Abstract

Efficient sampling from un-normalized target distributions is pivotal in scientific computing and machine learning. While neural samplers have demonstrated potential with a special emphasis on sampling efficiency, existing neural implicit samplers still have issues such as poor mode covering behavior, unstable training dynamics, and sub-optimal performances. To tackle these issues, in this paper, we introduce Denoising Fisher Training (DFT), a novel training approach for neural implicit samplers with theoretical guarantees. We frame the training problem as an objective of minimizing the Fisher divergence by deriving a tractable yet equivalent loss function, which marks a unique theoretical contribution to assessing the intractable Fisher divergences. DFT is empirically validated across diverse sampling benchmarks, including two-dimensional synthetic distribution, Bayesian logistic regression, and high-dimensional energy-based models (EBMs). Notably, in experiments with high-dimensional EBMs, our best one-step DFT neural sampler achieves results on par with MCMC methods with up to 200 sampling steps, leading to a substantially greater efficiency over 100 times higher. This result not only demonstrates the superior performance of DFT in handling complex high-dimensional sampling but also sheds light on efficient sampling methodologies across broader applications.