Learning finite-dimensional coding schemes with nonlinear reconstruction maps

TL;DR

This paper extends the theory of finite-dimensional lossy coding to nonlinear reconstruction maps, connecting it to generative modeling and providing generalization bounds, especially when using deep neural networks.

Contribution

It generalizes the Maurer--Pontil framework to nonlinear maps and establishes a link to generative modeling with new generalization bounds for finite data.

Findings

Connection established between coding schemes and optimal transportation theory.

Generalization bounds proved for learning coding schemes from finite samples.

Application demonstrated with deep neural network reconstruction maps.

Abstract

This paper generalizes the Maurer--Pontil framework of finite-dimensional lossy coding schemes to the setting where a high-dimensional random vector is mapped to an element of a compact set of latent representations in a lower-dimensional Euclidean space, and the reconstruction map belongs to a given class of nonlinear maps. Under this setup, which encompasses a broad class of unsupervised representation learning problems, we establish a connection to approximate generative modeling under structural constraints using the tools from the theory of optimal transportation. Next, we consider problem of learning a coding scheme on the basis of a finite collection of training samples and present generalization bounds that hold with high probability. We then illustrate the general theory in the setting where the reconstruction maps are implemented by deep neural nets.