Hyperspherical Variational Autoencoders Using Efficient Spherical Cauchy Distribution

TL;DR

This paper introduces a hyperspherical VAE using a spherical Cauchy distribution, offering a more natural, flexible, and numerically stable latent space representation for directional data compared to traditional methods.

Contribution

It presents the spCauchy distribution for VAEs, enabling stable training via M"obius transformations and efficient KL divergence computation, improving over vMF-based models.

Findings

spCauchy provides a more expressive latent space.

The model avoids numerical instabilities of vMF.

Training is stable and scalable with the new reparameterization.

Abstract

We propose a novel variational autoencoder (VAE) architecture that employs a spherical Cauchy (spCauchy) latent distribution. Unlike traditional Gaussian latent spaces or the widely used von Mises-Fisher (vMF) distribution, spCauchy provides a more natural hyperspherical representation of latent variables, better capturing directional data while maintaining flexibility. Its heavy-tailed nature prevents over-regularization, ensuring efficient latent space utilization while offering a more expressive representation. Additionally, spCauchy circumvents the numerical instabilities inherent to vMF, which arise from computing normalization constants involving Bessel functions. Instead, it enables a fully differentiable and efficient reparameterization trick via M\"obius transformations, allowing for stable and scalable training. The KL divergence can be computed through a rapidly converging power series, eliminating concerns of underflow or overflow associated with evaluation of ratios of hypergeometric functions. These properties make spCauchy a compelling alternative for VAEs, offering both theoretical advantages and practical efficiency in high-dimensional generative modeling.