Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces

TL;DR

This paper introduces S$^2$VAE, a geometry-focused variational autoencoder with hyperspherical latent distributions that better preserve 3D geometric information in compressed visual models.

Contribution

It proposes a novel hyperspherical latent encoding framework that enhances geometric preservation in vision transformers, outperforming Gaussian bottlenecks in high compression scenarios.

Findings

Hyperspherical latents outperform Gaussian bottlenecks in depth and pose tasks.

Geometry-aligned encoding improves performance under high compression.

Latent geometry is crucial for physically grounded visual models.

Abstract

Modern visual world modeling systems increasingly rely on high-capacity architectures and large-scale data to produce plausible motion, yet they often fail to preserve underlying 3D geometry or physically consistent camera dynamics. A key limitation lies not only in model capacity, but in the latent representations used to encode geometric structure. We propose S$^2$VAE, a geometry-first latent learning framework that focuses on compressing and representing the latent 3D state of a scene, including camera motion, depth, and point-level structure, rather than modeling appearance alone. Building on representations from a Visual Geometry Grounded Transformer (VGGT), we introduce a novel type of variational autoencoder using a product of Power Spherical latent distributions, explicitly enforcing hyperspherical structure in the bottleneck to preserve directional and geometric semantics under strong compression. Across depth estimation, camera pose recovery, and point cloud reconstruction, we show that geometry-aligned hyperspherical latents consistently outperform conventional Gaussian bottlenecks, particularly in high-compression regimes. Our results highlight latent geometry as a first-class design choice for physically grounded visual and world models.