SIRfyN: Single Image Relighting from your Neighbors

TL;DR

This paper introduces SIRfyN, a novel method for relighting a scene in a single image by estimating the illumination cone using similar scenes, without requiring ground truth data, enabling natural scene relighting and data augmentation.

Contribution

The paper presents a new theory and method to estimate scene lighting from similar scenes, avoiding the need for inverse graphics datasets or ground truth, for realistic image relighting.

Findings

Effectively erases and restores indoor shadows

Can steer light around a scene

Produces images suitable for data augmentation

Abstract

We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form a cone (the illumination cone). Novel theory shows that one can use similar scenes to estimate the different lightings that apply to a given scene, with bounded expected error. Our method exploits this theory to estimate a representation of the available lighting fields in the form of imputed generators of the illumination cone. Our procedure does not require expensive "inverse graphics" datasets, and sees no ground truth data of any kind. Qualitative evaluation suggests the method can erase and restore soft indoor shadows, and can "steer" light around a scene. We offer a summary quantitative evaluation of the method with a novel application of the FID. An extension of the FID allows per-generated-image evaluation. Furthermore, we offer qualitative evaluation with a user study, and show that our method produces images that can successfully be used for data augmentation.