Estimating Homogeneous Data-driven BRDF Parameters from a Reflectance Map under Known Natural Lighting

TL;DR

This paper presents a robust method for estimating homogeneous data-driven BRDF parameters from reflectance maps under natural lighting, leveraging material class similarities and Gaussian mixture models for improved accuracy.

Contribution

It introduces a novel approach combining Gaussian mixture models and non-linear optimization to estimate BRDF parameters from reflectance data under natural lighting conditions.

Findings

Effective estimation of BRDF parameters demonstrated on MERL database

Robustness shown under various natural lighting conditions

Proof-of-concept experiment validates real-world applicability

Abstract

In this paper we demonstrate robust estimation of the model parameters of a fully-linear data-driven BRDF model from a reflectance map under known natural lighting. To regularize the estimation of the model parameters, we leverage the reflectance similarities within a material class. We approximate the space of homogeneous BRDFs using a Gaussian mixture model, and assign a material class to each Gaussian in the mixture model. We formulate the estimation of the model parameters as a non-linear maximum a-posteriori optimization, and introduce a linear approximation that estimates a solution per material class from which the best solution is selected. We demonstrate the efficacy and robustness of our method using the MERL BRDF database under a variety of natural lighting conditions, and we provide a proof-of-concept real-world experiment.

Equations

$$\rho =Bw,$$

$${\rho }^{\prime }=B^{\prime }{w}^{\prime },$$

$$y({\omega }_o)={\int }_{\Omega }\rho ({\omega }_i,{\omega }_o)\cos ({\theta }_i)L({\omega }_i)d{\omega }_i,$$

$$y=Yw,$$

$$y_i={\int }_{\Omega }{b}_i({\omega }_i,{\omega }_o)\cos ({\theta }_i)L({\omega }_i)d{\omega }_i.$$

$$\zeta (y_i)\ne y_i^{\zeta }={\int }_{\Omega }\zeta (b_i({\omega }_i,{\omega }_o))\cos ({\theta }_i)L({\omega }_i)d{\omega }_i.$$

$$\underset{w}{\mathrm{argmax}}P(\rho |y).$$

$$\underset{w}{\mathrm{argmax}}\frac{P(y|\rho )P(\rho )}{P(y)}.$$

$$\underset{w}{\mathrm{argmin}}\left(\log P(y|\rho )+\log P(\rho )\right).$$

$$P(\rho )=\sum _{j=1}^k{\pi }_j\mathcal{N}(\rho |{\mu }_j,{\Sigma }_j),$$

$${\gamma }_j(b_i)$$

$$=$$

$${\pi }_j$$

$${\mu }_j$$

$${\Sigma }_j$$

$$\log P(B|\Theta )=\sum _i^n\log \sum _j^k{\pi }_j\mathcal{N}(\rho |{\mu }_j,{\Sigma }_j).$$

$$B=US{V}^T.$$

$$P(y|\rho )=\mathcal{N}(Yw-y|\mu ,\sigma ),$$

$$\underset{w}{\mathrm{argmin}}\left(\frac{||Yw-y|{|}^2}{{\sigma }^2}+\log \sum _j{\pi }_j\mathcal{N}({\hat{U}}^{T}Bw|{\mu }_j,{\Sigma }_j)\right).$$

$${\gamma }_j(b_i)\approx {\delta }_{i,m}.$$

$$\underset{w^{(j)}}{\mathrm{argmin}}\left(\log P(y|\rho ,j)+\log P(\rho |j)\right).$$

$$\log P(y|\rho ,j)=||Y^{(j)}{w}^{(j)}-y|{|}^2,$$

$$P(\rho |j)=\mathcal{N}(w^{(j)},{\mu }_j^{\prime },{\Sigma }_j^{\prime }),$$

$${\lambda }_j=\frac{\lambda ||y|{|}^2}{c_j},$$

$$\underset{w^{(j)}}{\mathrm{argmin}}\left(||Y^{(j)}{w}^{(j)}-y|{|}^2+{\lambda }_j\frac{||w^{(j)}-{\mu }_j^{\prime }|{|}^2}{{\Sigma }_j^2}\right).$$

Full text

1 Introduction

Data-driven appearance models [11, 12] express the Bidirectional Reflectance Distribution Function (BRDF) of a homogeneous material as a linear combination of a large set of measured “basis” BRDFs. The key assumption is that this large set of basis BRDFs covers the full space of BRDFs, and any BRDF in this space can be represented as convex combination of these basis BRDFs, thereby inheriting all the intricate reflectance details present in the measured basis BRDFs that can be difficult to model with analytical BRDF models. Compared to analytical BRDF models that require an expensive and fragile non-linear optimization to estimate the model parameters from reflectance measurements, data-driven BRDF models, by virtue of its linear nature, only require a linear least squares to estimate the model parameters. Recent advances have shown great promise in reconstructing a data-driven BRDF from very few measurements [13, 25]. However, these methods rely on controlled directional or point lighting. A key problem in generalizing prior methods to natural lighting is that these prior methods require a non-linear encoding (e.g., logarithmic) to compress the dynamic range of the the basis BRDFs in order to regularize the estimation of the model parameters. Such non-linear encoding can only be undone after linear parameter estimation if the observations consist of direct BRDF observations (i.e., a single view and a single light direction per observation). In contrast, observations under natural lighting are the result of an integration of the BRDF times lighting over all directions, and only linear transformations of the BRDF are transparent to this integration.

In this paper we aim to narrow the gap between inverse rendering with data-driven BRDF models and analytical BRDF models under natural lighting while retaining the robustness and simplicity of linear parameter estimation for data-driven models. We consider our work a first exploration in this direction that demonstrates that robust linear data-driven BRDF model parameter estimation under natural lighting is feasible, rather than introducing a practical and/or competitive method to current advanced inverse rendering methods that use an analytical BRDF models as a basis. To focus our exploration, we will a-priori assume that the natural lighting is known and that we have a full characterization of the material reflectance under this lighting condition in the form of a reflectance map [18].

We desire to retain the advantages of a linear parameter estimation process, and therefore avoid non-linear encoded basis BRDFs, and directly estimate the data-driven BRDF model parameters from unmodified basis BRDFs. To regularize the estimation of the model parameters from a reflectance map under natural lighting, we leverage the reflectance similarities between BRDFs in a material class. Intuitively, we expect that it is easier to express the BRDF as a combination of a small set of similar materials than from a large set of BRDFs that span a larger spectrum of more varied materials. We therefore, first approximate the space of homogeneous BRDFs with a Gaussian mixture model. Each normal distribution in the Gaussian mixture model represents a material class, and we assign each basis material to the class with the highest likelihood. We formulate the estimation of the model parameters as a maximum a-posteriori optimization that maximizes the likelihood that the model parameters explain the observations, as well as the likelihood that the model belongs to the material class. However, this formulation is highly non-linear and difficult to minimize. We therefore exploit the additional observation that in high dimensional spaces everything is distant, and approximate the maximum a-posteriori optimization by an efficient linear least squares approximation per material class. Finally, we select the most likely provisional least squares solution based on the maximum a-posteriori error.

We demonstrate the efficacy of our solution using the MERL BRDF database under a variety of natural lighting conditions. Furthermore, we provide a proof-of-concept real-world experiment to demonstrate that our results generalize beyond the ideal simulated experiments on the MERL BRDF database.

2 Related Work

We focus this discussion of prior work on the two key properties of our method: reflectance modeling under natural lighting, and appearance modeling with a data-driven reflectance model. We refer to the surveys of Dorsey et al. [5], and Weinmann and Klein [23] for an in-depth general overview of appearance modeling.

Reflectance Modeling under Natural Lighting

A first subset of methods models surface reflectance from multiple photographs under natural lighting. Oxholm and Nishino [15] model shape and homogeneous reflectance from multiple photographs under known natural lighting. Palma et al. [16], Dong et al. [4], and Zhou et al. [27] recover spatially-varying surface reflectance under unknown natural lighting from a dense sampling of multiple views or multiple rotations of a subject with known shape. Xia et al. [24] extended the method of Dong et al. [4] to model spatially-varying reflectance under unknown natural lighting and unknown shape. These model all rely on non-linear reflectance models and estimation processes. In contrast, we employ a linear data-driven BRDF model and rely on a linear estimation process.

A second subset of methods models surface reflectance from just a single photograph of an object under natural lighting. In seminal work, Ramamoorthi and Hanrahan [17] lay out a spherical harmonics framework for estimating general homogeneous reflectance functions modeled by a spherical harmonics expansion. Romeiro et al. [19, 20] model the homogeneous surface reflectance using a bivariate data-driven model from an object with known shape under known and unknown natural lighting respectively. Similarly, Lombardi et al. [10] also estimate natural lighting and homogeneous surface reflectance modeled by the DSBRDF reflectance model [14]. Finally, Barron and Malik [2] recover shape, lighting and spatially-varying albedo from a single photograph under unknown natural lighting. However, Barron and Malik only consider diffuse reflectance. Our method espouses the same overall goal as this second subset of methods. A reflectance map can potentially be obtained from a single observations of a convex object of known shape (e.g., sphere) or using the deep learning method of Rematas et al. [18]. However, we explicitely desire to recover a data-driven model [11] based on real-world measured reflectance.

A third subclass of methods relies on deep learning to infer reflectance properties under unknown natural lighting from a single image. Li et al. [7] and Ye et al. [26] estimate the parameters of an analytical BRDF model [22] for a spatially-varying material. Both Li et al. and Ye et al. focus on augmenting the training data with unlabeled photographs in order to reduce the number of required labeled training data (i.e., measured SVBRDFs). Li et al. [8] present a network structure and a novel post-processing step based on conditional random fields to estimate spatially-varying reflectance parameters for an analytical micro-facet BRDF model [21]. Finally, Li et al. [9] propose a cascading network structure to iteratively estimate and refine the shape and spatially-varying surface reflectance. All of the above methods express the surface reflectance using an analytical BRDF model. In contrast, we express the surface reflectance using a more expressive data-driven model, albeit limited to a homogeneous material and under known natural lighting.

Data-driven Reflectance Model

In seminal work, Matusik et al. [11] presented a data-driven BRDF model that expresses the surface reflectance as a weighted combination of a large set of measured BRDFs. To handle the large dynamic range between the specular peaks and the diffuse reflectance, a log-encoding is first applied to the measured basis BRDFs. Matusik et al. propose two models: a PCA based $45$D linear model, and non-linear, charting based, $15$D model. In follow up work, Matusik et al. [12] use the linear PCA model and show that $800$ well selected and controlled view-light direction pairs are sufficient for estimating the BRDF. Nielsen et al. [13] show that by adding a Tikhonov regularization to the estimation of a log-relative encoded linear data-driven model, a good BRDF estimate can be obtained from less than $20$ optimized and controlled view-light direction pairs, and for $5$ photographs of a sphere lit by optimized directional light sources. Xu et al. [25] build on the method of Nielsen et al., and show that with an improved error metric, a log-relative encoded linear data-driven model can be recovered from just $2$ near-field observations (photographs) under controlled directional lighting. All of the above methods estimate a data-driven BRDF from observations under directional lighting, and regularize the estimation using a non-linear encoding of the measured BRDFs. In contrast, our method uses a fully linear model and reconstructs the data-driven BRDF model from a reflectance map under uncontrolled known natural lighting.

3 Overview

Data-driven BRDF

The reflectance behavior of a homogeneous material is described by the bidirectional reflectance distribution function (BRDF) ${\rho}(\omega_{i},\omega_{o})$: a $4D$ function defined as the ratio of incident irradiance for an incident direction $\omega_{i}$ over the outgoing radiance for an outgoing direction $\omega_{o}$.

In this paper, we follow the data-driven BRDF model of Matusik et al. [12] that characterizes the BRDF ${\rho}$ as a linear combination of a large set of $n$ measured materials $b_{i},i\in[1,n]$. The underlying idea is that the set of measured BRDFs spans the space of BRDFs, and any material’s BRDF should lie in this space:

[TABLE]

where we stack the BRDF ${\rho}$ and basis BRDFs $b_{i}$ in a vector of length $p$, and form the matrix $B$ by stacking each basis vector in a column: $B=[b_{1},...,b_{n}]$. The model parameters are stacked in a vector $w$ of $n$ scalar weights. We directly use the BRDF parameterization of the MERL BRDF database [11], and $p=90\times 90\times 180$. Furthermore, similar as in Nielsen et al. [13], we consider each color channel of the $100$ MERL BRDFs as a basis BRDF, and thus $n=300$.

Due to the large dynamic range between specular peaks versus diffuse reflectance, prior work [11, 13, 25] has applied a non-linear compression function $\zeta$ to make the estimation of $w$ less sensitive to errors on the (large) specular peaks:

[TABLE]

where $B^{\prime}=[\zeta(b_{1}),...,\zeta(b_{n})]$. An expansion $\zeta^{-1}$ is applied to the compressed BRDF ${\rho}^{\prime}$ after computation of the weights. A common compression function is the logarithmic function, in which case Equation 2 becomes a homomorphic factorization.

Natural Lighting

Prior work relied on point sample measurements of ${\rho}$ for a set of incoming-outgoing direction pairs to estimate the weights $w$. In contrast, in this paper we aim to estimate the weights $w$ from an observation under natural lighting. Assuming the lighting $L$ is distant (i.e., it only depends on the incident direction $\omega_{i}=(\phi_{i},\theta_{i})$), and ignoring interreflections, we can formulate the observed radiance $y$ as:

[TABLE]

where $\cos(\theta_{i})$ is the foreshortening, and $\Omega$ is the upper hemisphere of incident directions. Due to linearity of light transport, we can express Equation 3 in terms of corresponding basis observations $y$:

[TABLE]

where the weights $w$ are the same as in Equation 1, and thus can be used to reconstruct ${\rho}$. The basis images $Y=[y_{0},...,y_{n}]$ are the observations of the measured basis BRDFs $b_{i}$ under the same conditions:

[TABLE]

Problem Statement

As noted before, the dynamic range compression function $\zeta$ is essential in obtaining good data-driven BRDF reconstructions, even in the case of a very dense point sampling of light and view directions [1]. However, this compression function cannot be used when linearly estimating the weights $w$ from observations under natural lighting. This can be seen by inserting Equation 2 in Equation 5:

[TABLE]

In other words, the non-linear compression of the observation is not equivalent to the observation under natural lighting of the non-linearly compressed BRDFs. While not a problem for the basis BRDFs $b_{i}$, since we can generate the corresponding images $y^{\zeta}_{i}$ with any rendering system directly from the non-linear encoded basis BRDFs $\zeta(b_{i})$, it is a problem for ${\rho}$, because we can only observe $y$ the resulting radiance of ${\rho}$ under natural lighting, not the reflected radiance of its non-linear compressed form $\zeta({\rho})$, and hence we do not have access to $y^{\zeta}$. Consequently, the key problem we aim to address in this paper is to find the data-driven weights $w$ from the observation $y$ without relying on a non-linear compression function $\zeta$ and/or a non-linear optimization procedure for estimating the weights $w$.

Maximum a-posteriori Optimization

Formally, our goal is to find the most likely weights $w$, relying on a linear estimation process, such that the conditional probability of the reconstructed data-driven homogeneous BRDF ${\rho}$ is maximized given a reflectance map $y$ under known natural lighting $L$:

[TABLE]

We will assume that the observations are in the form of a high dynamic range reflectance map (i.e., a full characterization of the reflectance radiance of a homogeneous BRDF for a fixed lighting condition). In the remainder of this paper, we will assume that the reflectance map is provided in the form of a visualization of a sphere under the target illumination.

Using Bayes’ theorem, we can formulate the maximum a-posteriori (MAP) estimation of $w$ as:

[TABLE]

Rewriting in terms of the log-likelihood, and noting that $P(y)$ is constant (i.e., the observation is given), we obtain:

[TABLE]

In order to solve this minimization problem, we need a model of the likelihood of the BRDF estimation ${\rho}$ (section 4), and a model for the conditional probability of the observation $y$ given the estimated BRDF ${\rho}$, and an efficient linear strategy for solving this minimization (section 5).

4 BRDF Likelihood Modeling

Gaussian Mixture Model

We propose to model the likelihood of BRDFs by a Gaussian mixture model (GMM):

[TABLE]

where $\pi_{j}$ are the mixing coefficients of the $j$-th normal distribution $\mathcal{N}$ with mean $\mu_{j}$ and covariance matrix $\Sigma_{j}$.

Expectation-Maximization

An effective method for computing the parameters $\Theta=(\pi,\mu,\Sigma)$ is the Expectation Maximization algorithm using the MERL BRDFs $b_{i}$ as observations. For this we define a latent variable $\gamma_{j}(b_{i})$ that indicates the likelihood of the $j$-th Gaussian given a MERL BRDF $b_{i}$:

[TABLE]

Expectation minimization iterates between estimating the latent variable $\gamma_{j}(b_{i})$ (E-step, Equation 13), and the model parameters (M-step):

[TABLE]

We iterate until the log-likelihood over the MERL BRDFs converges:

[TABLE]

To bootstrap the EM algorithm, we perform a standard k-mean clustering, and initialize $\pi_{j}$ as the ratio of assigned BRDFs to the $j$-th cluster over the total number of MERL BRDFs (i.e., $n$).

Curse of Dimensionality

A practical problem is that the number of observations $n$ is significantly lower than the dimensionality of the space (i.e., $p$).We therefore apply a singular value decomposition (SVD) to express the observations in a $n$ dimensional space $U$:

[TABLE]

However, this is still a $300$ dimensional space. A key issue is that even for a moderate number of dimensions any distance is very large, and thus the distance to the means $\mu_{j}$ are large too. Consequently, the likelihood of each Gaussian mixture (Equation 13) will always be very low and it can potentially cause numerical instabilities. To resolve this issue, we perform expectation maximization in a reduced space, and only keep the coefficients belonging to the $N$ largest singular values. In other words, we perform expectation maximization (i.e., soft clustering) on a projection to an $N$ dimensional subspace, and approximate the likelihood: $P({\rho})\approx P(\hat{U}^{T}{\rho})$, where $\hat{U}$ is the $N$ dimensional basis (i.e., the first $N$ vectors in $U$).

Discussion

We found that $N=4$ offers a good balance between accuracy and numerical stability. A second parameter that needs to be set is the number of Gaussian mixtures $K$. If the number of Gaussians is too low, then $P(\hat{U}^{T}{\rho})$ only offers a coarse approximation. However, we also found that for increasing number of $K$, the algorithm tends to subdivide the same Gaussian distribution, essentially overfitting to ’special case’ BRDFs (such as Steel which exhibits acquisition artifacts). In practice we found that $K=4$ offers a good approximation that nicely categorizes the materials in four recognizable distinct material classes: “diffuse and glossy” materials ($137$ materials), “plastics/phenolics” ($99$ materials), “metals” ($24$), and “specular plastics/paints” ($40$ materials); we determine membership to a material class by assigning the material to the material class with the maximum $\gamma_{j}(b_{i})$ likelihood. Figure 1 shows a plot of a $2$D multi-dimensional scaling of the $4$D projected coordinates of the MERL BRDFs, as well as a color-coding to indicate for which material class the material has the highest affinity. Note that even though the diffuse-like material class contains $137$ materials, the multi-dimensional scaling places them all close together. Please refer to the supplemental material for an exhaustive list of which material belongs to which material class.

5 Data-driven Model Estimation

MAP Estimation

We express the likelihood of the observation given an estimate of the BRDF as:

[TABLE]

where $\mu$ and $\Sigma$ is the expected mean error and standard deviation on the reconstructions, and $Yw$ is the rendering of the estimated BRDF under the target natural lighting. We assume that the mean error is close to zero ($\mu=0$), and $\sigma$ is proportional to the expected measurement error (e.g., camera noise).

Given the likelihood $P(\hat{U}^{T}{\rho})$ expressed by the Gaussian mixture model (Equation 10), we can then formulate the MAP estimation (Equation 9) as:

[TABLE]

The first term is the data term that indicates how well (a visualization of) the BRDF ${\rho}=Bw$ can explain the observation $y$, and the second term indicates how plausible the reconstructed BRDF ${\rho}$ (projected in the $4$ dimensional space $\hat{U}$) is.

However, directly solving for the BRDF weights $w$ using Equation 20 is not practical because of two key issues:

Non-linear: Equation 20 is highly non-linear and difficult to optimize due to the sum of the log-likelihoods in the second term. 2. 2.

Gaussian Mixture Model Accuracy for $P({\rho})\approx P(\hat{U}^{T}{\rho})$: We approximated the likelihood of the BRDF by a $4$ dimensional Gaussian mixture model. This reduction in dimensionality was necessary due to the curse of dimensionality. However, it also implicitly assumes that the BRDF lies not too far from the space of plausible BRDFs. Since the likelihood is only determined based on $4$ dimensions (and thus only regularizes these four), the other $296$ dimensions can be set to any value (including unreasonable values that result in an implausible BRDF).

Linear MAP Approximation

To alleviate the above two practical issues, we exploit the observation that the likelihood of a basis BRDF $b_{i}$ belonging to a material class $m$ is for most basis BRDFs equivalent to an indicator function:

[TABLE]

This implies that the overlap between the Gaussians in the Gaussian mixture model is limited. Armed with this observation, we therefore propose to compute a candidate BRDF for each material class $j\in[1,k]$:

[TABLE]

Given the set of candidate solutions $w^{\prime}=\{w^{(1)},..,w^{(k)}\}$, we then pick the best candidate that best reconstructs the BRDF.

Per-Material Class Linear Data Term

We define the data-term similarly as in the general non-linear case, except that we only use the basis BRDFs that belong to the same material class:

[TABLE]

where $Y^{(j)}$ is the set of observations that correspond to the basis BRDFs assigned to the $j$-th material class (i.e., the materials $b_{i}$ for which $\gamma_{j}(b_{i})$ is maximal).

Per-Material Class Linear Likelihood Term

We express the per-material class likelihood by a single Gaussian model. We directly compute this probability on the BRDF weights $w^{(j)}$:

[TABLE]

where: $\mu^{\prime}_{j}=(\frac{1}{c_{j}},...,\frac{1}{c_{j}})^{T}$, and $c_{j}$ is the number of basis BRDFs in the $j$-th material class. Note that $Y^{(j)}\mu^{\prime}_{j}$ is equivalent to the mean BRDF of the material class, and $\mu^{\prime}_{j}$ the corresponding coordinate in the j-th BRDF subspace.

Linear Least Squares Estimation

Both Equation 23 and (the log likelihood of) Equation 24 are quadratic terms that define a linear system in terms of $w$ that can be solved using a regular linear least squares. However, both terms can have a vastly different magnitude. The magnitude of the data-term depends on the error on the rendered image of the estimated BRDF. This image error depends on the resolution, the overall intensity of the lighting, and the reflectivity of the material. Similarly, the magnitude of the likelihood term depends on the number of basis BRDFs per material class. We therefore add a balancing term:

[TABLE]

where $||y||^{2}$ is the total squared pixel intensities in the observation. We expect that the overall intensity of the observation is directly proportional to the lighting intensity and reflectivity of the BRDF, and hence the overall scale of the image error. $\lambda$ is a user set constant that depends on the qualities of the lighting. An ill-conditioned lighting condition requires a larger $\lambda$ value (e.g., a low frequency lighting environment is ill-conditioned for estimating specular properties [17]). In practice we found that $\lambda=0.5$ works well for many lighting environments, and forms a good starting point for fine-tuning $\lambda$.

The final linear least squares is:

[TABLE]

Selection

Ideally, we would like to select the best candidate solution from $w^{\prime}$ by evaluating Equation 20. However, by a-priori assuming that a BRDF belongs to a material class $j$, it is possible that there is a significant mismatch between the target material and the material class. For example, attempting to model a mirror-like specular material using the diffuse material class is unlikely to produce a satisfactory result. Consequently, we cannot simply rely on the likelihood $P(\hat{U}^{T}{\rho})$ based on the 4 dimensional Gaussian mixture model to select the best solution from $w^{\prime}$ (i.e., the other $296$ dimensions can be arbitrarily wrong). We will therefore further exploit the observation of the limited overlap of the Gaussians in the mixture model, and approximate the solution per material class by enforcing that it lies in the convex hull of the subspace spanned by the BRDFs assigned to the material class, and only rely on Equation 19 to pick the best candidate from $w^{\prime}$. We ignore the standard deviation (i.e., $\sigma=1$) in Equation 19 as it only acts as a scale (in the log-likelihood) that does not affect the selection of the best reconstruction (i.e., minimum log-likelihood).

Color

Our discussion until now only considered monochrome BRDFs; we used all color channels from the MERL BRDFs as separate basis BRDFs. A straightforward strategy for estimating a non-monochrome BRDF with three color channels, would be to execute the estimation separately for each color channel, and combine the three reconstructed monochrome BRDF into a single RGB BRDF. However, it is possible that a solution from a different material classes $j$ is selected for each of the three color channels. Because the set of basis BRDFs for each material class are disjunct, there can be slight differences in the constructed BRDF shape for each color channel, which in turn can result in color artifacts in the combined BRDF. We circumvent this potential problem by combining the three color channels after obtaining the candidate BRDFs, and performing the selection on the RGB BRDF instead of each color channel separately. Hence, each color channel will be reconstructed with the same set of basis BRDFs.

Algorithm Summary

In summary, given a reflectance map $y$ under known natural lighting $L$, and given a user provided balance parameter $\lambda$, we compute the data-driven BRDF ${\rho}=Bw$ as:

We precompute the Gaussian mixture model using the EM algorithm detailed in section 4. Note, this precomputation only needs to happen once for the MERL BRDF database, and is independent of the lighting. 2. 2.

We precompute $Y$ by rendering a sphere with each basis BRDF $b_{i}$ under the natural lighting (Equation 5). This precomputation needs to happen for every lighting condition. 3. 3.

We compute the candidate solutions $w^{\prime}_{\{r,g,b\}}$ for each material class by solving the linear least squares in Equation 26 per color channel. 4. 4.

We combine the monochrome BRDFs to a 3-channel BRDF: $w^{\prime}=\{(w^{\prime}_{r,1},w^{\prime}_{g,1},w^{\prime}_{b,1}),...,(w^{\prime}_{r,k},w^{\prime}_{g,k},w^{\prime}_{b,k})\}$. 5. 5.

Finally, we select the candidate solution from $w^{\prime}$ that minimizes Equation 19.

6 Results

Experiment Setup

We demonstrate our method on simulated reflectance maps in order to fully control all parameters. We generate the reflectance maps under natural lighting, by rendering a sphere lit by a light probe [3] using Mitsuba [6]; as noted in section 3, we will directly use this rendered image as a representation of the reflectance map. All generated images are radiometrically linear, and we only tone map them for display. All results shown in this paper were tone mapped by a simple gamma $2.2$ correction and a virtual exposure (i.e., scale factor) of $1.0$; all pixel values above 1.0 and below 0.0 are clipped to the respective clipping values. We use the BRDFs in the MERL database [11] for generating reflectance maps. For each MERL BRDF, we compute a novel Gaussian mixture model on the $297$ remaining MERL BRDFs (i.e., we exclude the basis BRDF corresponding to any of the three color channels of the BRDF), and only use these $297$ MERL BRDFs for reconstruction. Consequently, any reconstruction of a BRDF from the MERL BRDF database is computed using a different set of basis BRDFs. As noted in the prior sections, we compute the Gaussian mixture model on a $N=4$ dimensional reduced space, and use $K=4$ Gaussians in the mixture model. All reconstructions are generated with a fixed balancing factor $\lambda=0.5$.

Reconstruction Results

Figure 2 shows reconstructions of $7$ selected materials under two different light probes (i.e., Eucalyptus Grove and Galileo’s Tomb). For each reconstruction (and the reference), we show a visualization of the reference/reconstructed BRDF under a natural lighting condition (i.e., Uffizi Gallery; different than the lighting condition under which the BRDF was reconstructed) and a directional light (i.e., a slice of the BRDF for a single incident direction for all outgoing directions). These results show that our method is able to reconstruct plausible BRDFs for a wide range of materials from a reflectance map under natural lighting. We refer to the supplemental material for the reconstructions under different natural lighting conditions for all MERL BRDFs.

Per-Material Class Reconstruction

Figure 3 illustrates, for a selection of $4$ materials, reconstructed under the Uffizi Gallery light probe, that the reconstructions per material class are different, and that depending on the material a different class’ reconstruction is selected. We show a visualization of the reference BRDF and the reconstructions per cluster under a natural lighting condition (i.e., Eucalyptus Grove) and a directional light. We also list the log-likelihood of the observation given the BRDF (Equation 19) below each cluster, and mark the final selected solution (i.e., minimum). For reference, we also show the linear least squares solution: $\operatorname*{argmin}_{w}||Yw-y||^{2}$. As expected this yields the lowest reconstruction error (since it explicitely optimizes for this). However, the linear least squares solution does not always yield a plausible result when visualized under a different lighting condition. This is not only clearly visible under the directional light source, but also under other natural lighting conditions other than the original observed lighting (e.g., the black spot in the center of the visualizations under the Eucalyptus Grove light probe for Steel and Red Metallic Paint). Furthermore, we observe that not all clusters’ reconstructions appear to be plausible. However, the selection process tends to pick the most plausible reconstruction.

Comparison: Single Material Class Reconstruction

To gain insight in the importance of reconstructing the BRDF per material class, we compare the reconstruction quality of the BRDF from a single material class to our multi-material class solution (Figure 4). Our results demonstrate that using a single material class improves on a naive linear least squares. However, our solution with multiple material classes outperforms the single material class case. Note that we optimized $\lambda$ for the single cluster case to produce an as optimal result given the lighting conditions. In this case we reconstructed the BRDF under Grace Cathedral lighting using a $\lambda=0.5$ for the single cluster case. Note, that the single material class reconstruction (Equation 26) is similar to Nielsen et al.’s [13] method, without applying a non-linear encoding of the BRDF. Additional minor differences are that Nielsen et al. subtract the median instead of the mean before computing the linear least squares and assume a unit standard deviation. Furthermore, the single class reconstruction is also similar to Romeiro et al.’s [19] method, using a linear data-driven BRDF model instead of the bivariate model. Since we a-priori assume a linear BRDF model, we want to explore the differences between the reconstruction methods, not the BRDF model representations.

In general, we found that overall our method outperforms a single material class reconstruction. The single material reconstruction tends to work equally well on phenolic and plastic materials as these are similar in BRDF shape to the mean material. However, the single material class reconstruction fails for diffuse and metal-like materials. While less strong than for the naive least squares, for diffuse materials we can observe a central “spike” visible under the directional lighting. For the metal-like materials, we typically observe strong ringing artifacts.

Captured Reflectance Map Validation

To validate our method on other materials than the MERL BRDF database, we performed the following proof-of-concept experiment. We acquired three spheres with different materials (i.e., Dense Orange Foam, Blue Plastic, and Dark Bronze) under two different natural lighting environments shown in the insets. Next, we estimate data-driven BRDF parameters under the indoor Chapel lighting, and rerender the sphere under the outdoor lighting. We mask out any measured reflectance values that deviate from the expected measurement conditions (e.g., the dimple on the Blue Plastic, and the near field reflection from the stand). As can be seen in Figure 5, the rerendered reflectance maps closely resemble the acquired reference maps. Note, the Dark Bronze material exhibits anisotropic reflectance which adversely impacts the reconstruction. Nevertheless, the reconstructed BRDF remains plausible. For reference, we also include a least squares data-driven BRDF reconstruction. In addition, we also show visualizations of the reconstructed BRDFs lit by a directional light source to better demonstrate the plausibility of the reconstructions.

Discussion

While our selection criterion does in the majority of cases select the best reconstruction from the different material classes, we found that in a few cases cases it does not select the best reconstruction, and a better reconstruction can be observed in another material class. Ideally, the selection criterion should not only include the data term Equation 19, but evaluate the full non-linear MAP estimation loss (Equation 20). We observe that for cases where our current selection criterion prefers a suboptimal solution, that the accompanying likelihood term $P(\hat{U}^{T}{\rho})$ is relatively large. However, a challenge is that the range of the data-term and the likelihood cover a different range due to: (a) the ommission of a standard deviation scale in the data-term, and (b) the dimension reduction in the likelihood term. Finding a good balancing term is non-trivial and an interesting avenue for future research.

We currently used a $\lambda$ balancing factor of $0.5$ for all our reconstructions. This $\lambda$ is a compromise to produce the best result over all materials. Despite the material class and scene dependent scale factor (Equation 25), we observe that this lambda terms tends to affect the “diffuse” and “plastics / phenolics” stronger, and the ”metals” and “specular plastics / paints” less. These latter two material classes exhibit not only a lower number of materials (for which we compensate), but we can also observe in Figure 1 that they are also spread out further. Consequently, the density of these material classes is significantly lower. This lower density implies that the material class is very diverse in BRDF types and that the MERL BRDF database does not densely sample these material types. Taking in account this density difference is another interesting avenue for future work. In general, we find that reconstructions from these material classes are less often selected.

Relation to Prior Work

Matusik et al. [11] showed that the log-encoded BRDF space can be accurately modeled by a $45$D linear subspace and a $15$D non-linear manifold. While a linear model is computationally more convenient, a non-linear model offers a tighter fit to the space of BRDFs, and consequently, it contains less implausible BRDFs. Our Gaussian mixture based model can be seen as a piecewise linear approximation of the non-linear manifold of BRDFs. In contrast to Matusik et al., we work directly on the space spanned by the basis BRDF (i.e., without log-encoding). However, as shown in Figure 1 this manifold is highly non-linear too. While less tight than a full $15$D non-linear model, our Gaussian mixture models strikes a balance between tightness and the ability to robustly identify the piecewise linear subspace to which the observations under natural lighting belong.

An implausible BRDF lies inside the linear subspace spanned by the linear model, but outside the non-linear BRDF manifold. Ideally, we would like to bias these implausible solution towards the non-linear manifold to obtain a more plausible solution. Tikhonov regularization biases the reconstruction towards a mean BRDF, assuming that the solution is more plausible when closer to the mean. However, such a regularization is only efficient if the modeled space resembles a hypersphere. Nielsen et al. [13] model the BRDF space as a hypersphere by scaling the PCA basis BRDFs by the singular values. However, as noted before, the BRDF space is highly non-linear and such a hypersphere is not a tight model. Intuitively, biasing the reconstruction of a diffuse material towards the mean or median BRDF is suboptimal; the mean or median BRDF contains a rough specular lobe. Consequently, it is possible that biasing pushes the solution towards a point away from the non-linear manifold. Our solution represents the space of BRDFs as a sum of (rescaled) hyperspheres: the per-material class linear likelihood term (Equation 24) biases the solution to the mean ($\mu^{\prime}_{j}$) of the local hypersphere (rescaled by $\Sigma^{\prime}_{j}$). Since each Gaussian subspace is more tight, biasing towards the mean has a lower likelihood of ending away from the non-linear manifold.

7 Conclusion

In this paper we presented a novel method for estimating the parameters of a fully linear data-driven BRDF model from a reflectance map under uncontrolled, but known, natural lighting. Our estimation method does not require any non-linear optimization, and only requires solving $4$ linear least squares problems. Our method requires modest precomputations: a Gaussian mixture model clustering for the basis BRDFs, and for each natural lighting conditions, renderings of each basis material. We demonstrated the accuracy and robustness of our method on the MERL BRDF database, and validated our method on real-world measurements.

For future work we would like to explore better selection criteria and a per-material class $\lambda_{j}$ density correction factor.

Acknowledgments

This work was supported in part by NSF grant IIS-1350323 and gifts from Google, Activision, and Nvidia.