Size-Noise Tradeoffs in Generative Networks

TL;DR

This paper explores the capabilities and limitations of generative networks in transforming input noise distributions into various target distributions, highlighting optimal constructions and efficient methods for distribution conversion.

Contribution

It introduces a novel construction for increasing noise distribution dimensionality in ReLU networks and provides efficient techniques for distribution transformations using polylogarithmic nodes.

Findings

ReLU networks can implement space-filling functions to increase distribution dimensionality.

Efficient methods exist for transforming between uniform and normal distributions.

High-dimensional distributions can be mapped to low-dimensional ones efficiently.

Abstract

This paper investigates the ability of generative networks to convert their input noise distributions into other distributions. Firstly, we demonstrate a construction that allows ReLU networks to increase the dimensionality of their noise distribution by implementing a "space-filling" function based on iterated tent maps. We show this construction is optimal by analyzing the number of affine pieces in functions computed by multivariate ReLU networks. Secondly, we provide efficient ways (using polylog $(1/\epsilon)$ nodes) for networks to pass between univariate uniform and normal distributions, using a Taylor series approximation and a binary search gadget for computing function inverses. Lastly, we indicate how high dimensional distributions can be efficiently transformed into low dimensional distributions.