Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture

TL;DR

This paper introduces new datasets and a physically consistent intrinsic decomposition-based network for single image relighting, enabling realistic lighting changes under arbitrary conditions and improving over prior methods.

Contribution

It provides two new datasets for relighting, proposes a two-stage intrinsic decomposition network with physical constraints, and demonstrates improved performance and animation capabilities.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Synthetic dataset pretraining enhances other methods.

Capable of generating animated relighting results.

Abstract

Single image scene relighting aims to generate a realistic new version of an input image so that it appears to be illuminated by a new target light condition. Although existing works have explored this problem from various perspectives, generating relit images under arbitrary light conditions remains highly challenging, and related datasets are scarce. Our work addresses this problem from both the dataset and methodological perspectives. We propose two new datasets: a synthetic dataset with the ground truth of intrinsic components and a real dataset collected under laboratory conditions. These datasets alleviate the scarcity of existing datasets. To incorporate physical consistency in the relighting pipeline, we establish a two-stage network based on intrinsic decomposition, giving outputs at intermediate steps, thereby introducing physical constraints. When the training set lacks ground truth for intrinsic decomposition, we introduce an unsupervised module to ensure that the intrinsic outputs are satisfactory. Our method outperforms the state-of-the-art methods in performance, as tested on both existing datasets and our newly developed datasets. Furthermore, pretraining our method or other prior methods using our synthetic dataset can enhance their performance on other datasets. Since our method can accommodate any light conditions, it is capable of producing animated results. The dataset, method, and videos are publicly available.