Stronger Semantic Encoders Can Harm Relighting Performance: Probing Visual Priors via Augmented Latent Intrinsics

TL;DR

This paper investigates the impact of semantic encoder strength on image relighting, revealing that stronger visual priors can hinder performance, and proposes ALI to balance semantic and photometric information for improved relighting results.

Contribution

It introduces Augmented Latent Intrinsics (ALI), a novel method that fuses semantic and photometric features to enhance relighting, addressing limitations of existing latent intrinsic representations.

Findings

Stronger semantic encoders can degrade relighting quality.

ALI improves relighting, especially on complex, specular materials.

The method achieves significant gains using only unlabeled real-world data.

Abstract

Image-to-image relighting requires representations that disentangle scene properties from illumination. Recent methods rely on latent intrinsic representations but remain under-constrained and often fail on challenging materials such as metal and glass. A natural hypothesis is that stronger pretrained visual priors should resolve these failures. We find the opposite: features from top-performing semantic encoders often degrade relighting quality, revealing a fundamental trade-off between semantic abstraction and photometric fidelity. We study this trade-off and introduce Augmented Latent Intrinsics (ALI), which balances semantic context and dense photometric structure by fusing features from a pixel-aligned visual encoder into a latent-intrinsic framework, together with a self-supervised refinement strategy to mitigate the scarcity of paired real-world data. Trained only on unlabeled real-world image pairs and paired with a dense, pixel-aligned visual prior, ALI achieves strong improvements in relighting, with the largest gains on complex, specular materials. Project page: https:\\augmented-latent-intrinsics.github.io