Latent Variables on Spheres for Autoencoders in High Dimensions

TL;DR

This paper introduces Spherical Auto-Encoder (SAE), a simple yet effective model that leverages high-dimensional geometry to improve latent space inference and sampling in variational autoencoders.

Contribution

The paper proposes SAE, a novel autoencoder with spherical normalization, addressing the dimensional dilemma in VAEs by exploiting high-dimensional sphere properties.

Findings

SAE outperforms traditional VAEs in sampling quality.

High-dimensional sphere properties make SAE agnostic to prior distributions.

Experimental results validate theoretical advantages of SAE.

Abstract

Variational Auto-Encoder (VAE) has been widely applied as a fundamental generative model in machine learning. For complex samples like imagery objects or scenes, however, VAE suffers from the dimensional dilemma between reconstruction precision that needs high-dimensional latent codes and probabilistic inference that favors a low-dimensional latent space. By virtue of high-dimensional geometry, we propose a very simple algorithm, called Spherical Auto-Encoder (SAE), completely different from existing VAEs to address the issue. SAE is in essence the vanilla autoencoder with spherical normalization on the latent space. We analyze the unique characteristics of random variables on spheres in high dimensions and argue that random variables on spheres are agnostic to various prior distributions and data modes when the dimension is sufficiently high. Therefore, SAE can harness a high-dimensional latent space to improve the inference precision of latent codes while maintain the property of stochastic sampling from priors. The experiments on sampling and inference validate our theoretical analysis and the superiority of SAE.