ReCap: Better Gaussian Relighting with Cross-Environment Captures

TL;DR

ReCap introduces a multi-task learning approach that leverages cross-environment captures to improve the physical accuracy of 3D object relighting, addressing the limitations of existing methods.

Contribution

It proposes a novel joint optimization of multiple lighting representations with shared material attributes to enhance relighting fidelity.

Findings

ReCap outperforms existing methods on an expanded relighting benchmark.

It achieves more physically consistent lighting reconstruction.

The approach improves robustness in material estimation.

Abstract

Accurate 3D objects relighting in diverse unseen environments is crucial for realistic virtual object placement. Due to the albedo-lighting ambiguity, existing methods often fall short in producing faithful relights. Without proper constraints, observed training views can be explained by numerous combinations of lighting and material attributes, lacking physical correspondence with the actual environment maps used for relighting. In this work, we present ReCap, treating cross-environment captures as multi-task target to provide the missing supervision that cuts through the entanglement. Specifically, ReCap jointly optimizes multiple lighting representations that share a common set of material attributes. This naturally harmonizes a coherent set of lighting representations around the mutual material attributes, exploiting commonalities and differences across varied object appearances. Such coherence enables physically sound lighting reconstruction and robust material estimation - both essential for accurate relighting. Together with a streamlined shading function and effective post-processing, ReCap outperforms all leading competitors on an expanded relighting benchmark.