Benign Autoencoders

TL;DR

This paper introduces the concept of benign autoencoders (BAE), a mathematical framework for optimal data encoding that enhances generative AI models by projecting data onto low-dimensional manifolds, improving stability and performance.

Contribution

It formalizes the BAE as an optimal encoder-decoder solution, connecting it to various AI models and demonstrating its effectiveness in finding low-dimensional, stable data representations.

Findings

BAE projects data onto an optimal low-dimensional manifold.

BAE improves discriminator performance under distribution shifts.

Compressing malignant data dimensions leads to smoother gradients.

Abstract

Recent progress in Generative Artificial Intelligence (AI) relies on efficient data representations, often featuring encoder-decoder architectures. We formalize the mathematical problem of finding the optimal encoder-decoder pair and characterize its solution, which we name the "benign autoencoder" (BAE). We prove that BAE projects data onto a manifold whose dimension is the optimal compressibility dimension of the generative problem. We highlight surprising connections between BAE and several recent developments in AI, such as conditional GANs, context encoders, stable diffusion, stacked autoencoders, and the learning capabilities of generative models. As an illustration, we show how BAE can find optimal, low-dimensional latent representations that improve the performance of a discriminator under a distribution shift. By compressing "malignant" data dimensions, BAE leads to smoother and more stable gradients.