Learning Physics-guided Face Relighting under Directional Light

TL;DR

This paper presents a deep learning approach for face relighting that decomposes images into intrinsic components and models non-diffuse effects, enabling realistic relighting in complex lighting and pose scenarios.

Contribution

The authors propose an end-to-end neural network that combines physics-based decomposition with residual correction for non-diffuse effects, trained on a new portrait dataset with diverse lighting and poses.

Findings

Achieves precise and believable face relighting results.

Generalizes well to complex illumination and challenging poses.

Outperforms existing methods in relighting accuracy.

Abstract

Relighting is an essential step in realistically transferring objects from a captured image into another environment. For example, authentic telepresence in Augmented Reality requires faces to be displayed and relit consistent with the observer's scene lighting. We investigate end-to-end deep learning architectures that both de-light and relight an image of a human face. Our model decomposes the input image into intrinsic components according to a diffuse physics-based image formation model. We enable non-diffuse effects including cast shadows and specular highlights by predicting a residual correction to the diffuse render. To train and evaluate our model, we collected a portrait database of 21 subjects with various expressions and poses. Each sample is captured in a controlled light stage setup with 32 individual light sources. Our method creates precise and believable relighting results and generalizes to complex illumination conditions and challenging poses, including when the subject is not looking straight at the camera.

Figures

Tables

Table 1. Table 1 : Loss selection. We explore the influence of different training losses and evaluation metrics on direct image-to-image translation (“pix2pix”) and our structured guidance approach (“ours”). For each class, we show validation scores for all pairwise combinations of 5 training losses (rows) and the same 5 evaluation metrics (columns). The best model for each evaluation metric is shown in bold.

M	Training Loss	Evaluation Metric
		$L_1$	$L_2$	LPIPS	DSSIM	MS-
						DSSIM
pix2pix	$L_1$	.0452	.0067	.2564	.1707	.1144
	$L_2$	.0516	.0082	.2663	.1911	.1369
	LPIPS	.0424	.0062	.1868	.1440	.0992
	DSSIM	.0406	.0055	.2138	.1378	.0930
	MS-DSSIM	.0422	.0058	.2358	.1547	.0913
ours	$L_1$	.0406	.0055	.2237	.1484	.0913
	$L_2$	.0415	.0056	.2302	.1547	.0953
	LPIPS	.0365	.0048	.1701	.1308	.0803
	DSSIM	.0362	.0045	.2008	.1270	.0793
	MS-DSSIM	.0410	.0055	.2165	.1470	.0910

Table 2. Table 2 : Quantitative evaluation. We show a quantitative comparison of our approach to baseline methods. Performance on the test set is reported under the assumption of both known (‘with’) and unknown (‘w/o’) source illumination. All models have been trained with the DSSIM loss.

L	Model	Evaluation Metric
		$L_1$	$L_2$	LPIPS	DSSIM	MS-
						DSSIM
with	SfSNet (R)	.0636	.0121	.2508	.1840	.1277
	pix2pix	.0668	.0144	.2430	.1832	.1328
	ours	.0609	.0123	.2144	.1618	.1138
w/o	SfSNet (P)	.1359	.0424	.4703	.3221	.3121
	pix2pix	.0815	.0189	.2783	.2076	.1623
	ours	.0684	.0142	.2273	.1763	.1316
	PMS	.0391	.0047	.1630	.1125	.0561

Equations

$$L_o({\omega }_o)={\int }_{{\omega }_i\in \Omega }f({\omega }_i,{\omega }_o)L_i({\omega }_i)⟨\mathbf{n},{\omega }_i⟩\mathrm{d}{\omega }_i,$$

$$a$$

$$s$$

$$f({\omega }_i,{\omega }_o)$$

$$L_o({\omega }_o)=as+{\int }_{{\omega }_i\in \Omega }r({\omega }_i,{\omega }_o)L_i({\omega }_i)⟨\mathbf{n},{\omega }_i⟩\mathrm{d}{\omega }_i.$$

$$L_o({\omega }_o)=(as+\tilde{r}(\widetilde{{\omega }_i},{\omega }_o))\cdot v({\tilde{\omega }}_i).$$

$$G(I_{\text{src}},{\ell }_{\text{src}},{\ell }_{\text{dst}})=I_{\text{dst}}.$$

Full text

Supplementary material can be found on our project page https://lvsn.github.io/face-relighting

1 Introduction

In recent years Augmented Reality (AR) has seen widespread interest across a variety of fields, including gaming, communication, and remote work. For an AR experience to be immersive, the virtual objects inserted in the environment should match the lighting conditions of their observed surroundings, even though they were originally captured under different lighting. This task, known as relighting, has a long history in computer vision with many seminal works paving the way for modern AR technologies [21, 3, 34, 2, 39].

Relighting is often represented as a physics-based, two-stage process. First, de-light the object in order to recover its intrinsic properties of reflectance, geometry, and lighting. Second, relight the object according to a desired target lighting. This implies an exact instantiation of the rendering equation  [17] operating on lighting and surface reflectance representations capable of capturing the true nature of the light-material-geometry interactions. In practice, errors occur due to imperfect parametric models or assumptions. One common approximation is to assume diffuse materials [2, 39]. Another approximation is smooth lighting, e.g. modeled as low-order spherical harmonics, which cannot produce hard shadows cast from point light sources like the sun. We consider the hard problem of relighting human faces, which are known for both their complex reflectance properties including subsurface scattering, view-dependent and spatially-varying reflectance, but also for our perceptual sensitivity to inaccurate rendering. Recent image-to-image translation approaches rely on deep learning architectures (e.g. [16]) that make no underlying structural assumption about the (re)lighting problem. Given enough representational capacity an end-to-end system can describe any underlying process, but is prone to large variance due to over-parameterization, and poor generalization due to physically implausible encodings. Test-time manipulation is also difficult with a semantically meaningless internal state. While this could potentially be alleviated with more training data, acquiring sufficient amounts is very time consuming.

Recent approaches have demonstrated that explicitly integrating physical processes in neural architectures is beneficial in terms of both robust estimates from limited data and increased generalization [43, 39, 25]. However, these approaches have focused on the de-lighting process, and use the simplified physical models for relighting that do not model non-diffuse effects such as cast shadows and specularities.

In this work, we bridge the gap between the expressiveness of a physically unconstrained end-to-end approach and the robustness of a physics-based approach. In particular, we consider relighting as an image-to-image translation problem and divide the relighting task into two distinct stages: a physics-based parametric rendering of estimated intrinsic components, and a physics-guided residual refinement. Our image formation model makes the assumption of directional light and diffuse materials. The subsequent refinement process is conditioned on the albedo, normals, and diffuse rendering, and dynamically accounts for shadows and any remaining non-diffuse phenomena.

We describe a neural architecture that combines the strengths of a physics-guided relighting process with the expressive representation of a deep neural network. Notably, our approach is end-to-end trained to simultaneously learn to both de-light and relight. We introduce a novel dataset of human faces under varying lighting conditions and poses, and demonstrate our approach can realistically relight complex non-diffuse materials like human faces. Our directional lighting representation does not require assumptions of smooth lighting environments and allows us to generalize to arbitrarily complex output lighting as a simple sum of point lights. To our knowledge, this is the first paper showing realistic relighting effects caused by strong directional lighting, such as sharp cast shadows, from a single input image.

2 Related work

Intrinsic images.

Intrinsic image decomposition [3] and the related problem of shape from shading [57] have inspired countless derived works. Of interest, [2] propose to simultaneously recover shape, illumination, reflectance and shading from a single image and rely on extensive priors to guide an inverse rendering optimization procedure. Other methods recover richer lighting representations in the form of environment maps given the known geometry [26]. More recent approaches rely on deep learning for the same task, for example using a combination of CNN and guided/bilateral filtering [31] or a pure end-to-end CNN approach [10] with the common problem of hard to come by training data. Available datasets may include only sparse relative reflectance judgements [4], or sparse shading annotations [20], which limits learning and quantitative evaluation.

While many previous works focus on lighting estimation from objects [2, 26, 12, 29] or even entire images [18, 54, 11, 15, 55], few papers explicitly focus on the relighting problem. Notably, [36] use a small number of images as input, and, more recently, [52] learn to determine which set of five light directions is optimal for relighting. Image-to-image translation [16] combined with novel multi-illumination datasets [30] has lately demonstrated promising results in full scene relighting.

The explicit handling of moving hard shadows in [9] and [35] is relevant. While both use multi-view inputs to relight outdoor scenes, our method works on a single input image to relight faces (our multi-view setup is only used to capture training data). Similar to our work, [53] regress to intrinsic components like albedo and normals, but their illumination model is spherical harmonics and therefore does not handle shadows. [38] recently proposed a residual appearance renderer which bears similarities to our learned residual in that it models non-Lambertian effects. Both of the latter works optimize for intrinsic decomposition, whereas we learn end-to-end relighting. Our intrinsic components are only used as a meaningful intermediate representation.

Face relighting.

Lighting estimation from face images often focuses on normalization for improving face recognition. For example, [50] use spherical harmonics (SH) to relight a face image, and [47] use a Markov random field to handle sharp shadows not modeled by low-frequency SH models. Other face modeling methods have exploited approximate lighting estimates to reconstruct the geometry [23, 45] or texture [24]. In computer graphics similar ideas have been proposed for face replacement [5, 8]. Low-frequency lighting estimation from a face has been explored in [41, 19, 40]. In contrast, [32] note that eyes reflect our surroundings and can be used to recover high frequency lighting. More closely related to our work, [7] learn the space of outdoor lighting using a deep autoencoder and combine this latent space with an inverse optimization framework to estimate lighting from a face. However, their work is restricted to outdoor lighting and cannot be used explicitly for relighting.

Of particular relevance to our work, neural face editing [43] and the related SfSNet [39] train CNNs to decompose a face image into surface normals, albedo, and SH lighting. These approaches also impose a loss on the intrinsic components, as well as a rendering loss which ensures that the combination of these components is similar to the input image. FRADA [22] revisited the idea of relighting for improving face recognition with face-specific 3D morphable models (similar to [43]), while we do not impose any face-specific templates. Single image portrait relighting [58] bypasses the need for decomposition, while still estimating the illumination to allow editing. In a similar line of work, [44] capture faces in a light stage using one light at a time, but then train using smoother illuminations from image based rendering which leads to artifacts when exposed to hard cast shadows or strong specularities. Recently, [28] also used light stage data and train to relight to directional lighting as we do. However, their network expects a pair of images captured under spherical gradient illumination at test time, which can only be captured in a light stage. The portrait lighting transfer approach of [42] directly transfers illumination from a reference portrait to an input photograph to create high-quality relit images, but fails when adding/removing non-diffuse effects.

3 Architecture

The following two sections first introduce an image formation process (Sec. 3.1) and then describe its integration into a physics-based relighting architecture (Sec. 3.2).

3.1 Image formation process

The image formation process describes the physics-inspired operations transforming the intrinsic properties of a 3D surface to a rendered output. The majority of physics-based works are based on specific instantiations of the rendering equation [17],

[TABLE]

where $\omega_{i},\omega_{o}$ are the incoming and outgoing light directions relative to the surface normal $\bf n$ at the surface point ${\bf X}_{j}$. $L_{i}(\omega_{i})$ and $L_{o}(\omega_{o})$ are the corresponding (ir)radiances, $f(\cdot,\cdot)$ is the BRDF describing the material’s reflectance properties, and $\langle{\bf n},\omega_{i}\rangle$ is the attenuating factor due to Lambert’s cosine law.

This model is often simplified further by assuming a diffuse decomposition into albedo $a\in\mathbb{R}$ and shading $s\in\mathbb{R}$,

[TABLE]

Non-diffuse effects.

A realistic relighting approach must relax modeling assumptions to allow complex reflectance properties such as subsurface scattering, transmission, polarization, etc., and if using 2 specularities. Unfortunately, learning a spatially varying BRDF model $f(\omega_{i},\omega_{o})$ based on a non-parametric representation is infeasible: assuming an image size of $512\times 768$ and a pixelwise discretization of the local half-angle space [27] would result in $1.7\text{\times}{10}^{12}$ parameters. Learning a low-dimensional representation in terms of semantic parameters [6] seems like a viable alternative but is still prone to overfitting and cannot account for light-material-interactions outside of its parametric space.

We propose a hybrid approach and decompose $f$ into two principled components, a diffuse albedo $a$ and a light-varying residual $r$:

[TABLE]

This turns 1 into

[TABLE]

For a light source with intensity $I(\omega_{i})$, we can identify $L_{i}(\omega_{i})=I(\omega_{i})v(\omega_{i})$, where $v\in\{0,1\}$ is the binary visibility of the light source. Under the assumption of a single directional light source from $\widetilde{\omega}_{i}$, we integrate over one point only, so if we further write $\widetilde{r}(\widetilde{\omega}_{i},\omega_{o})=r(\widetilde{\omega_{i}},\omega_{o})I(\widetilde{\omega}_{i})\langle{\bf n},\widetilde{\omega}_{i}\rangle$, we can re-formulate our rendering equation 1 to

[TABLE]

This will be the underlying image formation process in all subsequent sections. While $as$ captures much of the diffuse energy across the image according to an explicit generative model, the residual $\widetilde{r}(\widetilde{\omega}_{i},\omega_{o})$ accounts for physical effects outside of the space representable by 2 and is modeled as a neural network (akin to [38]). We do not impose any assumptions on $r({\widetilde{\omega}_{i},\omega_{o}})$, even allowing light subtraction, but do enforce $a$ to be close to the ground truth albedo of a diffuse model which we obtain from photometric stereo [51].

Discussion.

While directional lights are conceptually simple, they lead to challenging relighting problems. Our combination of an explicit diffuse rendering process and a non-diffuse residual (with implicit shading) serves several purposes: (1) Describing most of the image intensities with a physics-based model means the output image will be more consistent with the laws of physics; (2) Describing specular highlights as residuals alleviates learning with a CNN; (3) Leaving the residual unconstrained (up to ground truth guidance) allows us to model effects that are not explainable by the BRDF, such as subsurface scattering and indirect light; (4) Modeling visibility explicitly helps, because the simple diffuse model does not handle cast shadows. At the same time, expecting the residual to take care of shadow removal by resynthesis is much harder than just masking it.

3.2 Physics-guided relighting

Presented with an input image $I_{\text{src}}$ that was lit by an input illumination $\ell_{\text{src}}$, our goal is to learn a generator $G$, relighting $I_{\text{src}}$ according to a desired output illumination $\ell_{\text{dst}}$,

[TABLE]

At training time, we assume $\ell_{\text{src}}$ and $\ell_{\text{dst}}$ to be directional lights, which is known to be a particularly challenging instance of relighting and accurately matches our captured data (see Sec. 4). At test time, this is not a limitation, since we can easily fit a set of directional lights to an environment map to perform more complex relighting (see Sec. 6).

Our physics-guided approach to solving the relighting task consists of a recognition model inferring intrinsic components from observed images (de-lighting) and a generative model producing relit images from intrinsic components (relighting). While the recognition model takes the form of a traditional CNN, the generative model follows our image formation process (Sec. 3.1) and is represented by structured layers with clear physical meaning. In line with 6, we implement the latter as a two-stage process: (Stage 1) Using the desired target lighting, we compute shading from predicted normals and multiply the result with our albedo estimate to obtain a diffuse render; (Stage 2) Conditioned on all intrinsic states predicted in stage 1, we infer a residual image and a visibility map, which we combine with the diffuse render according to 6. An illustration of this pipeline is shown in Fig. 2. Since all its operations are differentiable and directly stacked, this allows us to learn the proposed model in an end-to-end fashion from input to relit result.

We introduce losses for all internal predictions, i.e., albedo, normals, shading, diffuse rendering, visibility, and residual. We emphasize the importance of using the right loss function and refer to Sec. 5.1 for a comprehensive study. In order to obtain the corresponding guidance during training, we use standard photometric stereo reconstruction [51].

4 Data

Our data comprises a diverse set of facial expressions captured under various lighting conditions.

4.1 Acquisition

We record our data in a calibrated multi-view light-stage consisting of $6$ stationary Sony PMW-F55 camcorders and a total of $32$ white LED lights. The cameras record linear HDR images at $2048\!\times\!1080$ / $60\text{\,}\mathrm{f}\mathrm{p}\mathrm{s}$ and are synchronized with a Blackmagic sync generator that also triggers the LED lights. We flash one LED per frame and instruct our subjects to hold a static expression for the full duration of an LED cycle ($32$ frames $\sim$ $0.53\text{\,}\mathrm{s}$). In order to remove captures with motion, we filter our data based on the difference of two fully lit shots before/after each cycle. For light calibration, we use a chrome sphere to recover directions and intensities in 3D [13] but subsequently express them with respect to each of the $6$ cameras, such that we obtain a total of $6\cdot 32=192$ different light directions/intensities for each image.

We record a total of $482$ sequences from $21$ subjects, resulting in $482\cdot 6\cdot 32\cdot 32=2{,}961{,}408$ relighting pairs. Each pair is formed using any one of the $32$ lights as input, and any one taken from the same sequence and same camera as output. We split them into $81\%$ ($17$ subjects) for training, $9.5\%$ ($2$ subjects) for validation and $9.5\%$ ($2$ subjects) for testing.111The split into training, validation and testing was done manually in an effort to balance the demographics of the subjects. See supplementary material for details. We did not ask the subjects to follow any specific protocol of facial expressions, besides being diverse, such that our evaluation on validation/test data is on both unseen expressions and unseen subjects.

After extraction of the raw data, we use photometric stereo (PMS) reconstruction [51] to separate the input images $I$ into albedo $A$, shading $S$ with corresponding normals, and non-diffuse residual images $R=I-A\odot S$ per frame.

4.2 Augmentation

Modern neural architectures are much better at interpolation than extrapolation. It is therefore critical to cover the space of valid light transports as well as possible. To this end, we perform a series of data augmentations steps in an attempt to establish strong correlations throughout the parametric relighting space: (1) We flip all training images along the horizontal and vertical axis, increasing the effective dataset size by a factor of $4$. Note that this also requires adaptation of the corresponding light directions and normals; (2) We perform a linear scaling ${\bf x}^{\prime}=s\cdot{\bf x}$, $s\sim\mathcal{U}_{[0.6,1.1]}$, of the images, shading, residuals and light intensities. In practice, we did not observe substantial benefits compared to training without scaling; (3) We randomly perturb the light calibration with Gaussian noise $n\sim\mathcal{N}(0,0.01^{2})$ to improve generalization and account for minimal calibration errors; (4) For quantitative results, we perform a spatial rescaling to $\frac{1}{8}$th of the original image resolution ($135\times 256$), train on random crops of size $128\times 128$ and test on center crops with the same resolution to have comparability with SfSNet. Qualitative results are generated by rescaling to $\frac{1}{2}$ of the original resolution ($540\times 1024$), trained on random crops of size $512\times 768$ and tested on center crops of that resolution.

5 Experiments

Our models were implemented using PyTorch [33] with a U-Net [37] generator and PatchGAN [14] discriminator (for the final relit image) based on the implementations provided by pix2pix [16]. The images in our dataset are camera RAW, represented as 16-bit linear RGB values nominally in the range $[0,1]$. There is under- and over-shoot headroom, but for training and evaluation we clamp them into this range and linearly transform to $[-1,1]$ as input into the network.

5.1 Evaluation metric

Quantitatively comparing the relit prediction $\hat{I}_{\text{dst}}$ of the generator against the ground truth $I_{\text{dst}}$ requires an appropriate error measure. We consider the $L_{1}$ and $L_{2}$ norms but recognize that they do not coincide with human perceptual response. We also consider the “Learned Perceptual Image Patch Similarity” (LPIPS) loss suggested by [56] using the distance of CNN-features pretrained on ImageNet. Another prevailing metric of image quality assessment is structural similarity (SSIM) [48] and its multi-scale variant (MS-SSIM) [49]. In our evaluation, we use the corresponding dissimilarity measure $\text{DSSIM}=\frac{1-\text{SSIM}}{2}$, and likewise for MS-SSIM, to consistently report errors.

When defining the loss function during training, the same choices of distance metrics are available. To densely evaluate their performance, we report in Table 1 the results of training all intrinsic layers with the same loss function from the options above. Surprisingly, we conclude that, for our task, using DSSIM for the training loss consistently leads to models which generalize better on the validation set using most of the error metrics. The only exception is evaluation using the LPIPS metric, which is better when also trained using this metric. Therefore, we chose the models trained on DSSIM for computing the final test results.

5.2 Baseline comparisons

We now provide quantitative and qualitative comparisons of our proposed architecture, to related work.

5.2.1 Baselines

Our baselines comprise the following set of related methods.

• PMS.

To understand the lower error bound of a diffuse model, we take albedo $A$ and shading $S$ from photometric stereo (PMS; [51]) and diffuse render via $A\odot S$. We note that this model has access to all light configurations at the same time, with the desired target illumination amongst them. Since this gives an unfair advantage, we do not highlight results for this model in Table 2.

• SfSNet (pretrained).

We take the pretrained network of SfSNet [39] and apply it to our data by using their decomposition into albedo and normals, but ignoring the output spherical harmonics estimate. Instead, we compute target shading as the dot product of $\ell_{\text{dst}}$ and normals to have a direct comparison with our assumption of directional light and present the result after diffuse rendering.

• SfSNet (retrained).

We retrain [39] on the calibrated PMS data and also provide the source illumination as input, to which our model has access as well. Compared to the pretrained model above, this baseline can be seen as a fairer comparison to SfSNet.

• Pix2pix.

The arguably simplest way to learn 7 from data is to instantiate $G$ as a traditional neural network consisting of a series of generic convolutional layers with no semantic meaning and no knowledge of the image formation process.We adapt the pix2pix translation GAN [16] to our use case by conditioning the generator on the input image as well as the source and target illumination. This ensures an objective comparison with our more structured model, which also has access to lighting information.

5.2.2 Evaluation

Qualitative evaluation.

We compiled a collection of qualitative results in Fig. 3. While the shading of SfSNet is smooth, it has a bias towards an albedo which probably resembles skin color in their training data and does not distinguish well between different skin tones and hair. As expected, retraining their model on our data leads to more accurate results. Still, due to the diffuse assumption, it looks flat compared to our more expressive model. It misses specularities and surface normals orthogonal to the light direction are missing ambient light from inter-reflections. The pix2pix model generates promising results, but its domain-agnostic architecture often leads to physically implausible artifacts, such as missing shadows. In comparison, the predictions of our proposed architecture are typically the most realistic, mainly due to its need to estimate a consistent albedo, as can be seen for example at the hair in the first row of Fig. 3. The last row shows a hand occluding the face, leading to strong cast shadows that have to be introduced/removed. Our model using intrinsic guidance gracefully handles that case.

While our data allows foreground masking computed from PMS, we show the full image predictions for better judgment. At test time, an off-the-shelf face matting approach, e.g. [46], could be used for cleaning the predictions.

We encourage the reader to look at more qualitative results of this type on our project page222https://lvsn.github.io/face-relighting, where we also show relighting under a moving target illumination.

Quantitative evaluation.

In Table 2 (first block), we analyze the quantitative performance of our model in the described scenario with known source illumination $\ell_{\text{src}}$. The test set comparison with SfSNet and the diffuse pix2pix baseline confirms the importance of our physics-based guidance and non-Lambertian residuals. An extension of our model without the assumption of known source illumination (second block) will be discussed in Sec. 6.1. For reference, the PMS reconstruction, restricted to a diffuse model but computed from multiple images, is also shown.

5.3 Additional qualitative comparisons

We provide more qualitative comparisons to the following recent portrait relighting approaches in Fig. 4.

The mass transport relighting [42] approach is different in that it defines the target lighting as that of another portrait. To match those conditions, we set the desired output to directly be the ground truth reference image. Despite these optimal conditions, [42] fails to generate specular highlights and cast shadows, which are well-captured by our technique.

Single image portrait relighting using an environment map is learned in [44]. Training images are produced by compositing multiple ‘one light at a time’ captures. As already discussed in [44], the method fails on strong light.

Finally, [58] learn to do deep portrait relighting using a spherical harmonics representation, which also handles smooth lighting exclusively, as can be seen in Fig. 4.

6 Extensions

We now demonstrate that our model successfully generalizes to different scenarios, including unknown input illumination, relighting with environment maps, and relighting images captured in the wild.

6.1 Relighting with unknown source illumination

While we cannot remove the need for the target illumination, information about the source illumination is already contained in the input image, allowing for implicit learning of $\ell_{\text{src}}$. To illustrate our model’s ability to extract these signals, we trained a version of our architecture without explicit access to the input lighting; these results, as well as a comparison to the corresponding baseline variants, are shown in Table 2 (second block) and Fig. 5. As expected, all models incur a small drop in performance compared to their counterparts with explicit knowledge. Nonetheless, our model without access to the source illumination achieves similar (and in some cases better) performance than the pix2pix model with access to the source illumination.

6.2 Relighting with environment maps

Directional light sources are a very general representation, and our approach easily allows for relighting with environment maps. While more principled approaches like importance sampling are available [1], for illustrative purposes here we simply sample environment maps by downscaling them to $64\times 32$ pixels and instantiating our relighting prediction with one light direction per pixel. Since light is additive, we then mix the resulting predictions according to their color and intensity. Examples of a relit scene with three environment maps are shown in Fig. 6, more results are contained in the supplementary material on our project page.

6.3 Relighting in the wild

To demonstrate generalization outside the domain of our lab-captured dataset, we conducted experiments using pictures taken in an office environment with a Canon EOS 5D Mark III and visualize results of relighting towards three target lights in Fig. 7.

We emphasize the practical difficulties of relighting those portraits, including unknown discrepancies in the imaging pipeline (camera sensor, illuminant color, image processing etc.) and approximation of the unknown source lighting. Since the portraits are taken under uncontrolled office lighting, this results in images which are diffusely lit by multiple input light sources. Although this violates our model assumption of a single directional light source, we run our model using an input light direction $\ell_{\text{src}}$ in the image that would simply light the portrait centrally. To compensate for different illumination colors in the input, we compute the mean of a $51\times 76$ center patch and apply a linear color transform towards our data distribution. The background is masked by hand, which could be automated [46]. Note that we can show neither quantitative nor qualitative comparisons to ground truth relit images since they do not exist.

7 Conclusion

We propose a method which learns to relight a face with strong directional lighting, accurately reproducing non-diffuse effects such as specularities and hard-cast shadows. We introduce a structured relighting architecture with semantic decomposition into, and subsequent re-rendering from intrinsic components. On our challenging light-stage dataset with directional light, the integration of an explicit generative rendering process and a non-diffuse neural refinement layer within an end-to-end architecture proved to be superior. We found that our model tends to produce shadows and albedo maps that are qualitatively closer to reality than all baselines. A more structured approach also has advantages beyond raw performance, including better interpretability/explainability and the possibility for direct manipulation/extraction of its semantically meaningful intermediate layers for downstream tasks. A comprehensive study of different losses and evaluation metrics highlighted the benefits of training on a perceptual loss over more traditional choices. We therefore believe our model to be useful in a wide range of face-centric and more general applications in, e.g., augmented reality.

Limitations and future work.

While our model generally deals well with cast shadows in the input image (see Fig. 4), results get worse when there is so little light that the camera mostly returns noise. Although a crude infilling for those pixels based on context can be learned, an interesting future direction would be to identify these pixels explicitly. A dedicated infilling method, conditioned on the properly relit parts of the image and other intrinsic layers, could be applied to them. To cancel ambient input illumination (as in Fig. 7), future work could experiment with taking a flash photo and subtracting a second non-flash photo (in linear color space).

Acknowledgments.

We thank Yannick Hold-Geoffroy and Marc-André Gardner for their valuable feedback, and Zhixin Shu and Tiancheng Sun for running their code on our dataset.