UniLight: A Unified Representation for Lighting

TL;DR

UniLight introduces a unified latent space for lighting that aligns multiple modalities like text, images, and environment maps, enabling improved cross-modal lighting understanding and manipulation in images.

Contribution

The paper proposes UniLight, a novel joint embedding space for diverse lighting representations, facilitating cross-modal transfer and manipulation in visual tasks.

Findings

Effective multi-modal alignment of lighting representations.

Improved performance in lighting-based retrieval and environment map generation.

Enables flexible lighting control in image synthesis.

Abstract

Lighting has a strong influence on visual appearance, yet understanding and representing lighting in images remains notoriously difficult. Various lighting representations exist, such as environment maps, irradiance, spherical harmonics, or text, but they are incompatible, which limits cross-modal transfer. We thus propose UniLight, a joint latent space as lighting representation, that unifies multiple modalities within a shared embedding. Modality-specific encoders for text, images, irradiance, and environment maps are trained contrastively to align their representations, with an auxiliary spherical-harmonics prediction task reinforcing directional understanding. Our multi-modal data pipeline enables large-scale training and evaluation across three tasks: lighting-based retrieval, environment-map generation, and lighting control in diffusion-based image synthesis. Experiments show that our representation captures consistent and transferable lighting features, enabling flexible manipulation across modalities.