Self-supervised Outdoor Scene Relighting

TL;DR

This paper introduces a self-supervised method for outdoor scene relighting that learns from internet images without supervision, effectively decomposing scenes and producing realistic relighting results that generalize well.

Contribution

It presents a novel self-supervised approach that decomposes images into albedo, geometry, and illumination, enabling relighting without synthetic training data.

Findings

Produces photo-realistic relighting results

Generalizes well to unseen outdoor scenes

Does not rely on accurate geometry estimation

Abstract

Outdoor scene relighting is a challenging problem that requires good understanding of the scene geometry, illumination and albedo. Current techniques are completely supervised, requiring high quality synthetic renderings to train a solution. Such renderings are synthesized using priors learned from limited data. In contrast, we propose a self-supervised approach for relighting. Our approach is trained only on corpora of images collected from the internet without any user-supervision. This virtually endless source of training data allows training a general relighting solution. Our approach first decomposes an image into its albedo, geometry and illumination. A novel relighting is then produced by modifying the illumination parameters. Our solution capture shadow using a dedicated shadow prediction map, and does not rely on accurate geometry estimation. We evaluate our technique subjectively and objectively using a new dataset with ground-truth relighting. Results show the ability of our technique to produce photo-realistic and physically plausible results, that generalizes to unseen scenes.