VENI: Variational Encoder for Natural Illumination

TL;DR

This paper introduces a rotation-equivariant variational autoencoder that models natural illumination on the sphere, improving interpolation and latent space behavior by leveraging novel SO(2)-equivariant layers and transformer architectures.

Contribution

It proposes a novel SO(2)-equivariant fully connected layer and a Vector Neuron Vision Transformer for modeling natural illumination without 2D projections.

Findings

Outperforms standard Vector Neurons in SO(2)-equivariant tasks.

Enables smoother interpolation in the latent space.

Provides a well-behaved latent space for illumination modeling.

Abstract

Inverse rendering is an ill-posed problem, but priors like illumination priors, can simplify it. Existing work either disregards the spherical and rotation-equivariant nature of illumination environments or does not provide a well-behaved latent space. We propose a rotation-equivariant variational autoencoder that models natural illumination on the sphere without relying on 2D projections. To preserve the SO(2)-equivariance of environment maps, we use a novel Vector Neuron Vision Transformer (VN-ViT) as encoder and a rotation-equivariant conditional neural field as decoder. In the encoder, we reduce the equivariance from SO(3) to SO(2) using a novel SO(2)-equivariant fully connected layer, an extension of Vector Neurons. We show that our SO(2)-equivariant fully connected layer outperforms standard Vector Neurons when used in our SO(2)-equivariant model. Compared to previous methods, our variational autoencoder enables smoother interpolation in latent space and offers a more well-behaved latent space.