GENs: Generative Encoding Networks

TL;DR

This paper introduces Generative Encoding Networks (GENs), a method that uses nonparametric density estimation to match latent space distributions to known targets, offering robustness and provable convergence, especially with limited data.

Contribution

The paper proposes a novel analytical approach for distribution matching in latent spaces using nonparametric density methods within autoencoders, improving robustness and theoretical guarantees.

Findings

Better performance with small training samples

Provable convergence properties

Advantages over adversarial methods

Abstract

Mapping data from and/or onto a known family of distributions has become an important topic in machine learning and data analysis. Deep generative models (e.g., generative adversarial networks ) have been used effectively to match known and unknown distributions. Nonetheless, when the form of the target distribution is known, analytical methods are advantageous in providing robust results with provable properties. In this paper, we propose and analyze the use of nonparametric density methods to estimate the Jensen-Shannon divergence for matching unknown data distributions to known target distributions, such Gaussian or mixtures of Gaussians, in latent spaces. This analytical method has several advantages: better behavior when training sample quantity is low, provable convergence properties, and relatively few parameters, which can be derived analytically. Using the proposed method, we enforce the latent representation of an autoencoder to match a target distribution in a learning framework that we call a {\em generative encoding network}. Here, we present the numerical methods; derive the expected distribution of the data in the latent space; evaluate the properties of the latent space, sample reconstruction, and generated samples; show the advantages over the adversarial counterpart; and demonstrate the application of the method in real world.