Explicit Density Approximation for Neural Implicit Samplers Using a Bernstein-Based Convex Divergence

TL;DR

This paper introduces dual-ISL, a convex, likelihood-free objective for training implicit generative models, leveraging Bernstein polynomial basis for explicit density approximation and improved convergence and stability.

Contribution

The work develops a novel convex discrepancy measure, dual-ISL, with a Bernstein polynomial basis for explicit density approximation, enhancing training stability and convergence in implicit generative models.

Findings

Dual-ISL converges faster than classical methods.

It produces smoother, more stable training.

It effectively prevents mode collapse.

Abstract

Rank-based statistical metrics, such as the invariant statistical loss (ISL), have recently emerged as robust and practically effective tools for training implicit generative models. In this work, we introduce dual-ISL, a novel likelihood-free objective for training implicit generative models that interchanges the roles of the target and model distributions in the ISL framework, yielding a convex optimization problem in the space of model densities. We prove that the resulting rank-based discrepancy $d_K$ is i) continuous under weak convergence and with respect to the $L^1$ norm, and ii) convex in its first argument-properties not shared by classical divergences such as KL or Wasserstein distances. Building on this, we develop a theoretical framework that interprets $d_K$ as an $L^2$-projection of the density ratio $q = p/\tilde p$ onto a Bernstein polynomial basis, from which we derive exact bounds on the truncation error, precise convergence rates, and a closed-form expression for the truncated density approximation. We further extend our analysis to the multivariate setting via random one-dimensional projections, defining a sliced dual-ISL divergence that retains both convexity and continuity. We empirically show that these theoretical advantages translate into practical ones. Specifically, across several benchmarks dual-ISL converges more rapidly, delivers markedly smoother and more stable training, and more effectively prevents mode collapse than classical ISL and other leading implicit generative methods-while also providing an explicit density approximation.