Local Relighting of Real Scenes

TL;DR

This paper introduces local relighting, a new task of modifying images by controlling visible light sources, and presents a training method using synthetic data and a new benchmark dataset.

Contribution

The paper proposes the first approach for local relighting trained without real paired data, using synthetic images from a GAN, and introduces the Lonoff dataset for benchmarking.

Findings

Our method outperforms GAN inversion baselines.

We demonstrate control over individual light sources.

The approach works without requiring real paired training data.

Abstract

We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image. This new task differs from the traditional image relighting problem, as it introduces the challenge of detecting light sources and inferring the pattern of light that emanates from them. We propose an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model. Concretely, we collect paired training images from a stylespace-manipulated GAN; then we use these images to train a conditional image-to-image model. To benchmark local relighting, we introduce Lonoff, a collection of 306 precisely aligned images taken in indoor spaces with different combinations of lights switched on. We show that our method significantly outperforms baseline methods based on GAN inversion. Finally, we demonstrate extensions of our method that control different light sources separately. We invite the community to tackle this new task of local relighting.