Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

TL;DR

This paper presents a comprehensive evaluation framework for scalable Bayesian deep learning methods in computer vision, comparing ensembling and MC-dropout, and finds ensembling to be more reliable for uncertainty estimation.

Contribution

The paper introduces a new evaluation framework for epistemic uncertainty estimation in deep learning, specifically designed for real-world computer vision applications, and provides the first extensive comparison of ensembling and MC-dropout.

Findings

Ensembling provides more reliable uncertainty estimates.

The evaluation framework tests robustness in real-world scenarios.

Ensembling outperforms MC-dropout in the comparison.

Abstract

While deep neural networks have become the go-to approach in computer vision, the vast majority of these models fail to properly capture the uncertainty inherent in their predictions. Estimating this predictive uncertainty can be crucial, for example in automotive applications. In Bayesian deep learning, predictive uncertainty is commonly decomposed into the distinct types of aleatoric and epistemic uncertainty. The former can be estimated by letting a neural network output the parameters of a certain probability distribution. Epistemic uncertainty estimation is a more challenging problem, and while different scalable methods recently have emerged, no extensive comparison has been performed in a real-world setting. We therefore accept this task and propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. Our proposed framework is specifically designed to test the robustness required in real-world computer vision applications. We also apply this framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates. Code is available at https://github.com/fregu856/evaluating_bdl.

Equations

$$\begin{array}{c}p(y|x,\theta )=\text{Cat}(y;s_{\theta }(x)),\\ s_{\theta }(x)=\text{Softmax}(f_{\theta }(x)).\end{array}$$

$$\begin{gathered}p(y|x,\theta)=\mathcal{N}\big{(}y;\mu_{\theta}(x),\,\sigma^{2}_{\theta}(x)\big{)},\\ f_{\theta}(x)=[\,\begin{matrix}\mu_{\theta}(x)&\log\sigma^{2}_{\theta}(x)\end{matrix}\,]^{\mathsf{T}}\in\mathbb{R}^{2}.\end{gathered}$$

$$\begin{array}{c}p(y^{\star }|x^{\star },\mathcal{D})=\int p(y^{\star }|x^{\star },\theta )p(\theta |\mathcal{D})d\theta \\ \approx \frac{1}{M}\sum _{i=1}^{M}p(y^{\star }|x^{\star },{\theta }^{(i)}),{\theta }^{(i)}\sim p(\theta |\mathcal{D}).\end{array}$$

$$\hat{p}(y^{\star }|x^{\star },\mathcal{D})\triangleq \frac{1}{M}\sum _{i=1}^{M}p(y^{\star }|x^{\star },{\theta }^{(i)}),{\theta }^{(i)}\sim q(\theta ),$$

$$\begin{gathered}y\sim\mathcal{N}\big{(}\mu(x),\sigma^{2}(x)\big{)},\\ \mu(x)=\sin(x),\quad\sigma(x)=0.15(1+e^{-x})^{-1}.\end{gathered}$$

$$\hat{p}(y^{\star }|x^{\star })\triangleq \frac{1}{M}\sum _{m=1}^{M}p(y^{\star }|x^{\star },{\hat{\theta }}^{(m)}).$$

$$\begin{array}{rl}& \hat{p}(y^{\star }|x^{\star },\mathcal{D})=\frac{1}{M}\sum _{i=1}^{M}p(y^{\star }|x^{\star },{\theta }^{(i)}),{\theta }^{(i)}\sim q(\theta ),\\ & \hat{p}(y^{\star }|x^{\star },\mathcal{D})=\frac{1}{M}\sum _{i=1}^M\mathcal{N}(y^{\star };{\mu }_{{\theta }^{(i)}}(x^{\star }),{\sigma }_{{\theta }^{(i)}}^2(x^{\star })),{\theta }^{(i)}\sim q(\theta ),\\ & \hat{p}(y^{\star }|x^{\star },\mathcal{D})\approx \mathcal{N}(y^{\star };\hat{\mu }(x^{\star }),{\hat{\sigma }}^2(x^{\star })),\end{array}$$

$$\hat{\mu}(x)=\frac{1}{M}\sum_{i=1}^{M}\mu_{\theta^{(i)}}(x),\quad\hat{\sigma}^{2}(x)=\frac{1}{M}\sum_{i=1}^{M}\bigg{(}\big{(}\mu_{\theta^{(i)}}(x)-\hat{\mu}(x)\big{)}^{2}+\sigma^{2}_{\theta^{(i)}}(x)\bigg{)},\quad\theta^{(i)}\sim q(\theta).$$

$$D_{\mathrm{KL}}(p_1\parallel p_2)=\log \frac{{\sigma }_2}{{\sigma }_1}+\frac{{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2}{2{\sigma }_2^2}-\frac{1}{2}.$$

$$D_{\mathrm{KL}}(q_1\parallel q_2)=\sum _{x\in \mathcal{X}}{q}_1(x)\log \frac{q_1(x)}{q_2(x)}.$$

$$L(\theta )=\frac{1}{N}\sum _{i=1}^N\frac{(y_i-\hat{\mu }(x_i){)}^2}{{\hat{\sigma }}^2(x_i)}+\log {\hat{\sigma }}^2(x_i)+\frac{1}{N}{\theta }^{\mathsf{T}}\theta .$$

$$L(\theta )=-\frac{1}{N}\sum _{i=1}^N\sum _{k=1}^C{y}_{i,k}\log \hat{s}(x_i{)}_k+\frac{1}{2N}{\theta }^{\mathsf{T}}\theta .$$

$${\theta }_{t+1}={\theta }_t-{\alpha }_t{\nabla }_{\theta }\tilde{U}({\theta }_t)+\sqrt{2{\alpha }_t}{ϵ}_t,$$

$${\alpha }_t={\alpha }_0(1-\frac{t}{T}{)}^{0.9},t=1,2,\dots ,T,$$

$$\begin{array}{rl}& {\theta }_{t+1}={\theta }_t+r_t,\\ & r_{t+1}=(1-\eta )r_t-{\alpha }_t{\nabla }_{\theta }\tilde{U}({\theta }_t)+\sqrt{2\eta {\alpha }_t}{ϵ}_t,\end{array}$$

$${\alpha }_t={\alpha }_0(1-\frac{t}{T}{)}^{0.9},t=1,2,\dots ,T,$$

Full text

1 Introduction

Deep Neural Networks (DNNs) have become the standard paradigm within most computer vision problems due to their astonishing predictive power compared to previous alternatives. Current applications include many safety-critical tasks, such as street-scene semantic segmentation [10, 6, 51], 3D object detection [42, 28] and depth completion [45, 32]. Since erroneous predictions can have disastrous consequences, such applications require an accurate measure of the predictive uncertainty. The vast majority of these DNN models do however fail to properly capture the uncertainty inherent in their predictions. They are thus not fully capable of the type of uncertainty-aware reasoning that is highly desired e.g. in automotive applications.

The approach of Bayesian deep learning aims to address this issue in a principled manner. Here, predictive uncertainty is commonly decomposed into two distinct types, which both should be captured by the learned DNN [12, 25]. Epistemic uncertainty accounts for uncertainty in the DNN model parameters, while aleatoric uncertainty captures inherent and irreducible data noise. Input-dependent aleatoric uncertainty about the target $y$ arises due to e.g. noise and ambiguities inherent in the input $x$. This is present for instance in street-scene semantic segmentation, where image pixels at object boundaries are inherently ambiguous, and in 3D object detection where the location of a distant object is less certain due to noise and limited sensor resolution. In many computer vision applications, this aleatoric uncertainty can be effectively estimated by letting a DNN directly output the parameters of a certain probability distribution, modeling the conditional distribution $p(y|x)$ of the target given the input. For classification tasks, a predictive categorical distribution is commonly realized by a softmax output layer, although recent work has also explored Dirichlet models [14, 41, 34]. For regression, Laplace and Gaussian models have been employed [23, 9, 25, 27].

Directly predicting the conditional distribution $p(y|x)$ with a DNN does however not capture epistemic uncertainty, as information about the uncertainty in the model parameters is disregarded. This often leads to highly confident predictions that are incorrect, especially for inputs $x$ that are not well-represented by the training distribution [17, 27]. For instance, a DNN can fail to generalize to unfamiliar weather conditions or environments in automotive applications, but still generate confident predictions. Reliable estimation of epistemic uncertainty is thus of great importance. However, this task has proven to be highly challenging, largely due to the vast dimensionality of the parameter space, which renders standard Bayesian inference approaches intractable. To tackle this problem, a wide variety of approximations have been explored [38, 22, 4, 48, 7, 21, 47], but only a small number have been demonstrated to be applicable even to the large-scale DNN models commonly employed in real-world computer vision tasks. Among such scalable methods, MC-dropout [12, 25, 24, 36] and ensembling [27, 9, 23] are clearly the most widely employed, due to their demonstrated effectiveness and simplicity. While scalable techniques for epistemic uncertainty estimation recently have emerged, the research community however lacks a common and comprehensive evaluation framework for such methods. Consequently, both researchers and practitioners are currently unable to properly assess and compare newly proposed methods. In this work, we therefore accept this task and set out to design exactly such an evaluation framework, aiming to benefit and inspire future research in the field.

Previous studies have provided only partial insight into the performance of different scalable methods for epistemic uncertainty estimation. Kendall and Gal [25] evaluated MC-dropout alone on the tasks of semantic segmentation and monocular depth regression, providing mainly qualitative results. Lakshminarayanan et al. [27] introduced ensembling as a non-Bayesian alternative and found it to generally outperform MC-dropout. Their experiments were however based on relatively small-scale models and datasets, limiting the real-world applicability. Ilg et al. [23] compared ensembling and MC-dropout on the task of optical-flow estimation, but only in terms of the AUSE metric which is a relative measure of the uncertainty estimation quality. While finding ensembling to be advantageous, their experiments were also limited to a fixed number ($M=8$) of ensemble members and MC-dropout forward passes, not allowing a completely fair comparison. Ovadia et al. [43] also fixed the number of ensemble members, and moreover only considered classification tasks. We improve upon this previous work and propose an evaluation framework that actually enables a conclusive ranking of the compared methods.

Contributions  We propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. The proposed framework is specifically designed to test the robustness required in real-world computer vision applications, and employs state-of-the-art DNN models on the tasks of depth completion (regression) and street-scene semantic segmentation (classification). It also employs a novel combination of quantitative evaluation metrics which explicitly measures the reliability and practical usefulness of estimated predictive uncertainties. We apply our proposed framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. This comparison demonstrates that ensembling consistently outperforms the highly popular MC-dropout method. Our work thus suggests that ensembling should be considered the new go-to approach, and encourages future research to understand and further improve its efficacy. Figure 1 shows example predictive uncertainty estimates generated by ensembling. Our framework can also directly be applied to compare other scalable methods, and we encourage external usage with publicly available code.

In our proposed framework, we predict the conditional distribution $p(y|x)$ in order to estimate input-dependent aleatoric uncertainty. The methods for epistemic uncertainty estimation are then compared by quantitatively evaluating the estimated predictive uncertainty in terms of the relative AUSE metric and the absolute measure of uncertainty calibration. Our evaluation is the first to include both these metrics, and furthermore we apply them to both regression and classification tasks. To provide a deeper and more fair analysis, we also study all metrics as functions of the number of samples $M$, enabling a highly informative comparison of the rate of improvement. Moreover, we simulate challenging real-world conditions found e.g. in automotive applications, where robustness to out-of-domain inputs is required to ensure safety, by training our models exclusively on synthetic data and evaluating the predictive uncertainty on real-world data. By analyzing this important domain shift problem, we significantly increase the practical applicability of our evaluation. We also complement our real-world analysis with experiments on illustrative toy regression and classification problems. Lastly, to demonstrate the evaluation rigor necessary to achieve a conclusive comparison, we repeat each experiment multiple times and report results together with the observed variation.

2 Predictive Uncertainty Estimation using Bayesian Deep Learning

DNNs have been shown to excel at a wide variety of supervised machine learning problems, where the task is to predict a target value $y\in\mathcal{Y}$ given an input $x\in\mathcal{X}$. In computer vision, the input space $\mathcal{X}$ often corresponds to the space of images. For classification problems, the target space $\mathcal{Y}$ consists of a finite set of $C$ classes, while a regression problem is characterized by a continuous target space, e.g. $\mathcal{Y}~{}=~{}\mathbb{R}^{K}$. For our purpose, a DNN is defined as a function $f_{\theta}:\mathcal{X}\rightarrow\mathcal{U}$, parameterized by $\theta\in\mathbb{R}^{P}$, that maps an input $x\in\mathcal{X}$ to an output $f_{\theta}(x)\in\mathcal{U}$. Next, we cover alternatives for estimating both the aleatoric and epistemic uncertainty in the predictions of DNN models.

Aleatoric Uncertainty  In classification problems, aleatoric uncertainty is commonly captured by predicting a categorical distribution $p(y|x,\theta)$. This is implemented by letting the DNN predict logit scores $f_{\theta}(x)\in\mathbb{R}^{C}$, which are then normalized by a Softmax function,

[TABLE]

Given a training set of i.i.d. sample pairs $\mathcal{D}=\{X,Y\}=\{(x_{i},y_{i})\}_{i=1}^{N}$, $(x_{i},y_{i})\sim p(x,y)$, the data likelihood is obtained as $p(Y|X,\theta)=\prod_{i=1}^{N}p(y_{i}|x_{i},\theta)$. The maximum-likelihood estimate of the model parameters, $\hat{\theta}_{\mathrm{MLE}}$, is obtained by minimizing the negative log-likelihood $-\sum_{i}\log p(y_{i}|x_{i},\theta)$. For the Categorical model (1), this is equivalent to minimizing the well-known cross-entropy loss. At test time, the trained model predicts the distribution $p(y^{\star}|x^{\star},\hat{\theta}_{\mathrm{MLE}})$ over the target class variable $y^{\star}$, given a test input $x^{\star}$. These DNN models are thus able to capture input-dependent aleatoric uncertainty, by outputting less confident predictions for inherently ambiguous cases.

In regression, the most common approach is to let the DNN directly predict targets, $y^{\star}=f_{\hat{\theta}}(x^{\star})$. The parameters $\hat{\theta}$ are learned by minimizing e.g. the $L^{2}$ or $L^{1}$ loss over the training dataset [42, 28]. However, such direct regression does not model aleatoric uncertainty. Instead, recent work [23, 25, 27] has explored predicting the distribution $p(y|x,\theta)$, similar to the classification case above. For instance, $p(y|x,\theta)$ can be parameterized by a Gaussian distribution [9, 27], giving the following model in the 1D case,

[TABLE]

Here, the DNN predicts the mean $\mu_{\theta}(x)$ and variance $\sigma^{2}_{\theta}(x)$ of the target $y$. The variance is naturally interpreted as a measure of input-dependent aleatoric uncertainty. As in classification, the model parameters $\theta$ are learned by minimizing the negative log-likelihood $-\sum_{i}\log p(y_{i}|x_{i},\theta)$.

Epistemic Uncertainty  While the above models can capture aleatoric uncertainty, stemming from the data, they are agnostic to the uncertainty in the model parameters $\theta$. A principled means to estimate this epistemic uncertainty is to perform Bayesian inference. The aim is to utilize the posterior distribution $p(\theta|\mathcal{D})$, which is obtained from the data likelihood and a chosen prior $p(\theta)$ by applying Bayes’ theorem. The uncertainty in the parameters $\theta$ is then marginalized out to obtain the predictive posterior distribution,

[TABLE]

Here, the generally intractable integral in (3) is approximated using $M$ Monte Carlo samples $\theta^{(i)}$, ideally drawn from the posterior. In practice however, obtaining samples from the true posterior $p(\theta|\mathcal{D})$ is virtually impossible, requiring an approximate posterior $q(\theta)\approx p(\theta|\mathcal{D})$ to be used. We thus obtain the approximate predictive posterior as,

[TABLE]

which enables us to estimate both aleatoric and epistemic uncertainty of the prediction. The quality of the approximation (4) depends on the number of samples $M$ and the method employed for generating $q(\theta)$. Prior work on such approximate Bayesian inference methods is discussed in Section 3. For the Categorical model (1), $\hat{p}(y^{\star}|x^{\star},\mathcal{D})=\textrm{Cat}(y^{\star};\hat{s}(x^{\star}))$, $\hat{s}(x^{\star})=\frac{1}{M}\sum_{i=1}^{M}s_{\theta^{(i)}}(x^{\star})$. For the Gaussian model (2), $\hat{p}(y^{\star}|x^{\star},\mathcal{D})$ is a uniformly weighted mixture of Gaussian distributions. We approximate this mixture with a single Gaussian, see Appendix A for details.

Illustrative Example  To visualize and provide intuition for the problem of predictive uncertainty estimation with DNNs, we consider the problem of regressing a sinusoid corrupted by input-dependent Gaussian noise,

[TABLE]

Training data $\{(x_{i},y_{i})\}_{i=1}^{1000}$ is only given in the interval $[-3,3]$, see Figure 2(a). A DNN trained to directly predict the target $y$ is able to accurately regress the mean for $x^{\star}~{}\in~{}[-3,3]$, see Figure 2(b). However, this model does not capture any notion of uncertainty. A corresponding Gaussian DNN model (2) trained via maximum-likelihood obtains a predictive distribution that closely matches the ground truth for $x^{\star}\in[-3,3]$, see Figure 2(c). While correctly accounting for aleatoric uncertainty, this model generates overly confident predictions for inputs $|x^{\star}|>3$ not seen during training. Finally, the Gaussian DNN model trained via approximate Bayesian inference (4), with a prior distribution $p(\theta)~{}=~{}\mathcal{N}(0,I_{P})$ and $M=1\thinspace 000$ samples obtained via Hamiltonian Monte Carlo [39], is additionally able to predict more reasonable uncertainties in the region with no available training data, see Figure 2(d).

3 Related Work

Here, we discuss prior work on approximate Bayesian inference. We also note that ensembling, which is often considered a non-Bayesian alternative, in fact can naturally be viewed as an approximate Bayesian inference method.

Approximate Bayesian Inference  The method employed for approximating the posterior $q(\theta)\approx p(\theta|\mathcal{D})=p(Y|X,\theta)p(\theta)/p(Y|X)$ is a crucial choice, determining the quality of the approximate predictive posterior $\hat{p}(y^{\star}|x^{\star},\mathcal{D})$ in (4). There exists two main paradigms for constructing $q(\theta)$, the first one being Markov chain Monte Carlo (MCMC) methods. Here, samples $\theta^{(i)}$ approximately distributed according to the posterior are obtained by simulating a Markov chain with $p(\theta|\mathcal{D})$ as its stationary distribution. For DNNs, this approach was pioneered by Neal [38], who employed Hamiltonian Monte Carlo (HMC) on small feed-forward neural networks. HMC entails performing Metropolis-Hastings [35, 19] updates using Hamiltonian dynamics based on the potential energy $U(\theta)\triangleq-\log p(Y|X,\theta)p(\theta)$. To date, it is considered a “gold standard” method for approximate Bayesian inference, but does not scale to large DNNs or large-scale datasets. Therefore, Stochastic Gradient MCMC (SG-MCMC) [33] methods have been explored, in which stochastic gradients are utilized in place of their full-data counterparts. SG-MCMC variants include Stochastic Gradient Langevin Dynamics (SGLD) [48], where samples $\theta^{(i)}$ are collected from the parameter trajectory given by the update equation $\theta_{t+1}=\theta_{t}-\alpha_{t}\nabla_{\theta}\widetilde{U}(\theta_{t})+\sqrt{2\alpha_{t}}\epsilon_{t}$, where $\epsilon_{t}\sim\mathcal{N}(0,1)$ and $\nabla_{\theta}\widetilde{U}(\theta)$ is the stochastic gradient of $U(\theta)$. Save for the noise term $\sqrt{2\alpha_{t}}\epsilon_{t}$, this update is identical to the conventional SGD update when minimizing the maximum-a-posteriori (MAP) objective $-\log p(Y|X,\theta)p(\theta)$. Similarly, Stochastic Gradient HMC (SGHMC) [7] corresponds to SGD with momentum injected with properly scaled noise. Given a limited computational budget, SG-MCMC methods can however struggle to explore the high-dimensional and highly multi-modal posteriors of large DNNs. To mitigate this problem, Zhang et al. [53] proposed to use a cyclical stepsize schedule to help escaping local modes in $p(\theta|\mathcal{D})$.

The second paradigm is that of Variational Inference (VI) [22, 2, 16, 4]. Here, a distribution $q_{\phi}(\theta)$ parameterized by variational parameters $\phi$ is explicitly chosen, and the best possible approximation is found by minimizing the Kullback-Leibler (KL) divergence with respect to the true posterior $p(\theta|\mathcal{D})$. While principled, VI methods generally require sophisticated implementations, especially for more expressive variational distributions $q_{\phi}(\theta)$ [30, 31, 52]. A particularly simple and scalable method is MC-dropout [13]. It entails using dropout [44] also at test time, which can be interpreted as performing VI with a Bernoulli variational distribution [13, 24, 36]. The approximate predictive posterior $\hat{p}(y^{\star}|x^{\star},\mathcal{D})$ in (4) is obtained by performing $M$ stochastic forward passes on the same input.

Ensembling  Lakshminarayanan et al. [27] created a parametric model $p(y|x,\theta)$ of the conditional distribution using a DNN $f_{\theta}$, and learned multiple point estimates $\{\hat{\theta}^{(m)}\}_{m=1}^{M}$ by repeatedly minimizing the MLE objective $-\log p(Y|X,\theta)$ with random initialization. They then averaged over the corresponding parametric models to obtain the following predictive distribution,

[TABLE]

The authors considered this a non-Bayesian alternative to predictive uncertainty estimation. However, since the point estimates $\{\hat{\theta}^{(m)}\}_{m=1}^{M}$ always can be seen as samples from some distribution $\hat{q}(\theta)$, we note that (6) is virtually identical to the approximate predictive posterior in (4). Ensembling can thus naturally be viewed as approximate Bayesian inference, where the level of approximation is determined by how well the implicit sampling distribution $\hat{q}(\theta)$ approximates the posterior $p(\theta|\mathcal{D})$. Ideally, $\{\hat{\theta}^{(m)}\}_{m=1}^{M}$ should be distributed exactly according to $p(\theta|\mathcal{D})~{}\propto~{}p(Y|X,\theta)p(\theta)$. Since $p(Y|X,\theta)$ is highly multi-modal in the parameter space for DNNs [1, 8], so is $p(\theta|\mathcal{D})$. By minimizing $-\log p(Y|X,\theta)$ multiple times, starting from randomly chosen initial points, we are likely to find different local optima. Ensembling can thus generate a compact set of samples $\{\hat{\theta}^{(m)}\}_{m=1}^{M}$ that, even for small values of $M$, captures this important aspect of multi-modality in $p(\theta|\mathcal{D})$.

4 Experiments

We conduct experiments both on illustrative toy regression and classification problems (Section 4.1), and on the real-world computer vision tasks of depth completion (Section 4.2) and street-scene semantic segmentation (Section 4.3). Our evaluation is motivated by real-world conditions found e.g. in automotive applications, where robustness to varying environments and weather conditions is required to ensure safety. Since images captured in these different circumstances could all represent distinctly different regions of the vast input image space, it is infeasible to ensure that all encountered inputs will be well-represented by the training data. Thus, we argue that robustness to out-of-domain inputs is crucial in such applications. To simulate these challenging conditions and test the robustness required for such real-world scenarios, we train all models on synthetic data and evaluate them on real-world data. To improve rigour of our evaluation, we repeat each experiment multiple times and report results together with the observed variation. A more detailed description of all results are found in the Appendix (Appendix B.3, C.2, D.2). All experiments are implemented in PyTorch [40].

4.1 Illustrative Toy Problems

We first present results on illustrative toy problems to gain insights into how ensembling and MC-dropout fare against other approximate Bayesian inference methods. For regression, we conduct experiments on the 1D problem defined in (5) and visualized in Figure 2. We use the Gaussian model (2) with two separate feed-forward neural networks outputting $\mu_{\theta}(x)$ and $\log\sigma^{2}_{\theta}(x)$. We evaluate the methods by quantitatively measuring how well the obtained predictive distributions approximate that of the “gold standard” HMC [39] with $M\!=\!1\thinspace 000$ samples and prior $p(\theta)=\mathcal{N}(0,I_{P})$. We thus consider the predictive distribution visualized in Figure 2(d) ground truth, and take as our metric the KL divergence $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ with respect to this target distribution $p_{\mathrm{HMC}}$. For classification, we conduct experiments on the binary classification problem in Figure 3. The true data generator is visualized in Figure 3(a), where red and blue represents the two classes. The training dataset contains $520$ examples of each class, and is visualized in Figure 3(b). We use the Categorical model (1) with a feed-forward neural network. As for regression, we quantitatively measure how well the obtained predictive distributions approximate that of HMC, which is visualized in Figure 3(c). Further details are provided in Appendix B.

Results  A comparison of ensembling, MC-dropout, SGLD and SGHMC in terms of $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ is found in Figure 5. The Adam optimizer [26] is here used for both ensembling and MC-dropout. We observe that ensembling consistently outperforms the compared methods, and MC-dropout in particular. Even compared to SG-MCMC variants such as SGLD and SGHMC, ensembling thus provides a better approximation to the MCMC method HMC. This result is qualitatively supported by visualized predictive distributions found in Appendix B.5. Example predictive distributions for ensembling and MC-dropout with $M\!=\!16$ are shown in Figure 4. We observe that ensembling provides reasonable approximations to HMC even for quite small values of $M$, especially compared to MC-dropout.

4.2 Depth Completion

In depth completion, we are given an image $x_{\mathrm{img}}\in\mathbb{R}^{h\times w\times 3}$ from a forward-facing camera and an associated sparse depth map $x_{\mathrm{sparse}}\in\mathbb{R}^{h\times w}$. Only non-zero pixels of $x_{\mathrm{sparse}}$ correspond to LiDAR depth measurements, projected onto the image plane. The goal is to predict a dense depth map $y\in\mathbb{R}^{h\times w}$ of the scene. We utilize the KITTI depth completion [15, 45] and Virtual KITTI [11] datasets. KITTI depth completion contains more than $80\thinspace 000$ images $x_{\mathrm{img}}$, sparse depth maps $x_{\mathrm{sparse}}$ and semi-dense target maps $y$. There are $1\thinspace 000$ selected validation examples, which we use for evaluation. Only about $4\%$ of the pixels in $x_{\mathrm{sparse}}$ are non-zero and thus correspond to depth measurements. The semi-dense target maps are created by merging the LiDAR scans from $11$ consecutive frames into one, producing $y$ in which roughly $30\%$ of the pixels are non-zero. Virtual KITTI contains synthetic images $x_{\mathrm{img}}$ and dense depth maps $x_{\mathrm{dense}}$ extracted from $5$ driving sequences in a virtual world. It contains $2\thinspace 126$ unique frames, of which there are $10$ different versions corresponding to various simulated weather and lighting conditions. We take sequence $0002$ as our validation set, leaving a total of $18\thinspace 930$ training examples. We create targets $y$ for training by setting all pixels in $x_{\mathrm{dense}}$ corresponding to a depth $>80$m to [math], and then also randomly sample $5\%$ of the remaining non-zero pixels uniformly to create $x_{\mathrm{sparse}}$. We use the DNN model presented by Ma et al. [32]. The inputs $x_{\mathrm{img}}$, $x_{\mathrm{sparse}}$ are separately processed by initial convolutional layers, concatenated and fed to an encoder-decoder architecture based on ResNet34 [20]. We employ the Gaussian model (2) by duplicating the final layer, outputting both $\mu\in\mathbb{R}^{h\times w}$ and $\log\sigma^{2}\in\mathbb{R}^{h\times w}$ instead of only the predicted depth map $\hat{y}\in\mathbb{R}^{h\times w}$. We also employ the same basic training procedure as Ma et al. [32] to train all our models, see Appendix C.1 for details. For the MC-dropout comparison, we take inspiration from Kendall et al. [24] and place a dropout layer with drop probability $p=0.5$ after the three last encoder blocks and the four first decoder blocks.

Evaluation Metrics  We evaluate the methods in terms of the Area Under the Sparsification Error curve (AUSE) metric, as introduced by Ilg et al. [23]. AUSE is a relative measure of the uncertainty estimation quality, comparing the ordering of predictions induced by the estimated predictive uncertainty (sorted from least to most uncertain) with the “oracle” ordering in terms of the true prediction error. The metric thus reveals how well the estimated uncertainty can be used to sort predictions from worst (large true prediction error) to best (small prediction error). We compute AUSE in terms of Root Mean Squared Error (RMSE) and based on all pixels in the entire evaluation dataset. A perfect AUSE score can however be achieved even if the true predictive uncertainty is consistently underestimated. As an absolute measure of uncertainty estimation quality, we therefore also evaluate the methods in terms of calibration [5, 46]. In classification, the Expected Calibration Error (ECE) [17, 37] is a standard metric used to evaluate calibration. A well-calibrated model should then produce classification confidences which match the observed prediction accuracy, meaning that the model is not over-confident (outputting highly confident predictions which are incorrect), nor over-conservative. We here employ a metric that can be considered a natural generalization of ECE to the regression setting. Since our models output the mean $\mu\in\mathbb{R}$ and variance $\sigma^{2}\in\mathbb{R}$ of a Gaussian distribution for each pixel, we can construct pixel-wise prediction intervals $\mu\pm\Phi^{-1}(\frac{p+1}{2})\sigma$ of confidence level $p\in]0,1[$, where $\Phi$ is the CDF of the standard normal distribution. When computing the proportion of pixels for which the prediction interval covers the true target $y\in\mathbb{R}$, we expect this value, denoted $\hat{p}$, to equal $p\in]0,1[$ for a perfectly calibrated model. We compute the absolute error with respect to perfect calibration, $|p-\hat{p}|$, for $100$ values of $p\in]0,1[$ and use the area under this curve as our metric, which we call Area Under the Calibration Error curve (AUCE). Lastly, we also evaluate in terms of the standard RMSE metric.

Results  A comparison of ensembling and MC-dropout in terms of AUSE, AUCE and RMSE on the KITTI depth completion validation dataset is found in Figure 6. We observe in Figure 6(a) that ensembling consistently outperforms MC-dropout in terms of AUSE. However, the curves decrease as a function of $M$ in a similar manner. Sparsification plots and sparsification error curves are found in Appendix C.3. A ranking of the methods can be more readily conducted based on Figure 6(b), where we observe a clearly improving trend as $M$ increases for ensembling, whereas MC-dropout gets progressively worse. This result is qualitatively supported by the calibration plots found in Appendix C.3 and Figure 7. Note that $M=1$ corresponds to the baseline of only estimating aleatoric uncertainty.

4.3 Street-Scene Semantic Segmentation

In this task, we are given an image $x\in\mathbb{R}^{h\times w\times 3}$ from a forward-facing camera. The goal is to predict $y$ of size $h\times w$, in which each pixel is assigned to one of $C$ different class labels (road, sidewalk, car, etc.). We utilize the popular Cityscapes [10] and recent Synscapes [49] datasets. Cityscapes contains $5\thinspace 000$ finely annotated images, mainly collected in various German cities. The annotations includes $30$ class labels, but only $C=19$ are used in the training of models. Its validation set contains 500 examples, which we use for evaluation. Synscapes contains $25\thinspace 000$ synthetic images, all captured in virtual urban environments. To match the size of Cityscapes, we randomly select $2\thinspace 975$ of these for training and $500$ for validation. The images are annotated with the same class labels as Cityscapes. We use the DeepLabv3 DNN model presented by Chen et al. [6]. The input image $x$ is processed by a ResNet101 [20], outputting a feature map of stride 8. The feature map is further processed by an ASPP module and a $1\times 1$ convolutional layer, outputting logits at 1/8 of the original resolution. These are then upsampled to image resolution using bilinear interpolation. The conventional Categorical model (1) is thus used for each pixel. We base our implementation on the one by Yuan and Wang [51], and also follow the same basic training procedure, see Appendix D.1 for details. For reference, the model obtains an mIoU [29] of $76.04\%$ when trained on Cityscapes and evaluated on its validation set. For the MC-dropout comparison, we take inspiration from Mukhoti and Gal [36] and place a dropout layer with $p=0.5$ after the four last ResNet blocks.

Evaluation Metrics  As for depth completion, we evaluate the methods in terms of the AUSE metric. In this classification setting, we compare the “oracle” ordering of predictions with the one induced by the predictive entropy. We compute AUSE in terms of Brier score and based on all pixels in the evaluation dataset. We also evaluate in terms of calibration by the ECE metric [17, 37]. All predictions are here partitioned into $L$ bins based on the maximum assigned confidence. For each bin, the difference between the average predicted confidence and the actual accuracy is then computed, and ECE is obtained as the weighted average of these differences. We use $L=10$ bins of equal size.

Results  A comparison of ensembling and MC-dropout in terms of AUSE, ECE and mIoU on the Cityscapes validation dataset is found in Figure 8. We observe that the metrics clearly improve as functions of $M$ for both ensembling and MC-dropout, demonstrating the importance of epistemic uncertainty estimation. The rate of improvement is generally greater for ensembling. For ECE, we observe in Figure 8(b) a drastic improvement for ensembling as $M$ is increased, followed by a distinct plateau. According to the condensed reliability diagrams in Appendix D.3, this corresponds to a transition from clear model over-confidence to slight over-conservatism. For MC-dropout, the corresponding diagrams suggest a stagnation while the model still is somewhat over-confident. Example reliability diagrams for $M=16$ are shown in Figure 9, in which this over-confidence for MC-dropout can be observed. Note that the relatively low mIoU scores reported in Figure 8(c), obtained by models trained exclusively on Synscapes, are expected [49] and caused by the intentionally challenging domain gap between synthetic and real-world data.

5 Discussion & Conclusion

We proposed a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. The proposed framework is specifically designed to test the robustness required in real-world computer vision applications. We applied our proposed framework and provided the first properly extensive and conclusive comparison of ensembling and MC-dropout, the results of which demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates. We attribute the success of ensembling to its ability, due to the random initialization, to capture the important aspect of multi-modality present in the posterior distribution $p(\theta|\mathcal{D})$ of DNNs. MC-dropout has a large design-space compared to ensembling, and while careful tuning of MC-dropout potentially could close the performance gap on individual tasks, the simplicity and general applicability of ensembling must be considered key strengths. The main drawback of both methods is the computational cost at test time that grows linearly with $M$, limiting real-time applicability. Here, future work includes exploring the effect of model pruning techniques [50, 18] on predictive uncertainty quality. For ensembling, sharing early stages of the DNN among ensemble members is also an interesting future direction. A weakness of ensembling is the additional training required, which also scales linearly with $M$. The training of different ensemble members can however be performed in parallel, making it less of an issue in practice given appropriate computing infrastructure. In conclusion, our work suggests that ensembling should be considered the new go-to method for scalable epistemic uncertainty estimation.

Acknowledgments  This research was supported by the Swedish Foundation for Strategic Research via the project ASSEMBLE and by the Swedish Research Council via the project Learning flexible models for nonlinear dynamics.

Supplementary Material

This is the supplementary material for Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision. It consists of Appendix A-D.

Appendix A Approximating a Mixture of Gaussian Distributions

For the Gaussian model (2), $\hat{p}(y^{\star}|x^{\star},\mathcal{D})$ in (4) is a uniformly weighted mixture of Gaussian distributions. We approximate this mixture with a single Gaussian parameterized by the mixture mean and variance:

[TABLE]

[TABLE]

Appendix B Illustrative Toy Problems

In this appendix, further details on the illustrative toy problems experiments (Section 4.1) are provided.

B.1 Experimental Setup

Figure 5(a) (regression) shows $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ computed on $[-7,7]$. All training data was given in $[-3,3]$.

Figure 5(b) (classification) shows $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ computed on the region $-6\leq x_{1}\leq 6$, $-6\leq x_{2}\leq 6$. All training data was given in the region $0\leq x_{1}\leq 3$, $-3\leq x_{2}\leq 3$.

For regression, $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ is computed using the formula for KL divergence between two Gaussian distributions $p_{1}(x)=\mathcal{N}(x;\mu_{1},\sigma_{1}^{2})$, $p_{2}(x)=\mathcal{N}(x;\mu_{2},\sigma_{2}^{2})$:

[TABLE]

For classification, $D_{\mathrm{KL}}(p~{}\parallel~{}p_{\mathrm{HMC}})$ is computed using the formula for KL divergence between two discrete distributions $q_{1}(x)$, $q_{2}(x)$:

[TABLE]

For both regression and classification, HMC with prior $p(\theta)~{}=~{}\mathcal{N}(0,I_{P})$ and $M=1\thinspace 000$ samples is implemented using Pyro [3]. Specifically, we use pyro.infer.mcmc.MCMC with pyro.infer.mcmc.NUTS as kernel, $\text{num}\_\text{samples}=1\thinspace 000$ and $\text{warmup}\_\text{steps}=1\thinspace 000$.

B.2 Implementation Details

For regression, we use the Gaussian model (2) with two separate feed-forward neural networks outputting $\mu_{\theta}(x)$ and $\log\sigma^{2}_{\theta}(x)$. Both neural networks have $2$ hidden layers of size $10$.

For classification, we use the Categorical model (1) with a feed-forward neural network with $2$ hidden layers of size $10$.

For the MC-dropout comparison, we place a dropout layer after the first hidden layer of each neural network. For regression, we use a drop probability $p=0.2$. For classification, we use $p=0.1$.

For ensembling, we train all ensemble models for $150$ epochs with the Adam optimizer, a batch size of $32$ and a fixed learning rate of $0.001$.

For MC-dropout, we train models for $300$ epochs with the Adam optimizer, a batch size of $32$ and a fixed learning rate of $0.001$.

For ensembling and MC-dropout, we minimize the MAP objective $-\log p(Y|X,\theta)p(\theta)$. In our case where the model parameters $\theta\in\mathbb{R}^{P}$ and $p(\theta)~{}=~{}\mathcal{N}(0,I_{P})$, this corresponds to the following loss for regression:

[TABLE]

For classification, where $y_{i}=[\,\begin{matrix}y_{i,1}\dots y_{i,C}\end{matrix}\,]^{\mathsf{T}}$ (one-hot encoded) and $\hat{s}(x_{i})=[\,\begin{matrix}\hat{s}(x_{i})_{1}\dots\hat{s}(x_{i})_{C}\end{matrix}\,]^{\mathsf{T}}$ is the Softmax output, it corresponds to the following loss:

[TABLE]

For SGLD, we extract samples from the parameter trajectory given by the update equation:

[TABLE]

where $\epsilon_{t}\sim\mathcal{N}(0,1)$, $\nabla_{\theta}\widetilde{U}(\theta)$ is the stochastic gradient of $U(\theta)=-\log p(Y|X,\theta)p(\theta)$ and $\alpha_{t}$ is the stepsize. We run it for a total number of steps corresponding to $256\cdot 150$ epochs with a batch size of $32$. The stepsize $\alpha_{t}$ is decayed according to:

[TABLE]

where $T$ is the total number of steps, $\alpha_{0}=0.01$ (the initial stepsize) for regression and $\alpha_{0}=0.05$ for classification. $M\in\{8,16,32,64,128,256\}$ samples are extracted starting at step $t=\text{int}(0.75T)$, ending at step $t=T$ and spread out evenly between.

For SGHMC, we extract samples from the parameter trajectory given by the update equation:

[TABLE]

where $\epsilon_{t}\sim\mathcal{N}(0,1)$, $\nabla_{\theta}\widetilde{U}(\theta)$ is the stochastic gradient of $U(\theta)=-\log p(Y|X,\theta)p(\theta)$, $\alpha_{t}$ is the stepsize and $\eta=0.1$. We run it for a total number of steps corresponding to $256\cdot 150$ epochs with a batch size of $32$. The stepsize $\alpha_{t}$ is decayed according to:

[TABLE]

where $T$ is the total number of steps, $\alpha_{0}=0.001$ (the initial stepsize) for regression and $\alpha_{0}=0.01$ for classification. $M\in\{8,16,32,64,128,256\}$ samples are extracted starting at step $t=\text{int}(0.75T)$, ending at step $t=T$ and spread out evenly between.

For all models, we randomly initialize the parameters $\theta$ using the default initializer in PyTorch.

B.3 Description of Results

The results in Figure 5(a), 5(b) were obtained in the following way:

• •

Ensembling: $1024$ models were trained using the same training procedure, the mean and standard deviation was computed based on $1024/M$ unique sets of models for $M~{}\in~{}\{8,16,32,64,128,256\}$.

• •

MC-dropout: $10$ models were trained using the same training procedure, based on which the mean and standard deviation was computed.

• •

SGLD: $6$ models were trained using the same training procedure, based on which the mean and standard deviation was computed.

• •

SGHMC: $6$ models were trained using the same training procedure, based on which the mean and standard deviation was computed.

B.4 Additional Results

Figure 10 and Figure 11 show the same comparison as Figure 5(a), 5(b), but using SGD and SGD with momentum for ensembling and MC-dropout, respectively. We observe that ensembling consistently outperforms the compared methods for classification, but that SGLD and SGHMC has better performance for regression in these cases. SGLD and SGHMC are however trained for 256 times longer than each ensemble model, complicating the comparison somewhat. If SGLD and SGHMC instead are trained for just 64 times longer than each ensemble model, we observe in Figure 12 that they are consistently outperformed by ensembling.

For MC-dropout using Adam, we also varied the drop probability $p$ and chose the best performing variant. These results are found in Figure 13, in which * marks the chosen variant.

B.5 Qualitative Results

Here, we show visualizations of predictive distributions obtained by the different methods. Figure 14, 18 for ensembling, Figure 15, 19 for MC-dropout, Figure 16, 20 for SGLD, and Figure 17, 21 for SGHMC.

Appendix C Depth Completion

In this appendix, further details on the depth completion experiments (Section 4.2) are provided.

C.1 Training Details

For both ensembling and MC-dropout, we train all models for $40\thinspace 000$ steps with the Adam optimizer, a batch size of $4$, a fixed learning rate of $10^{-5}$ and weight decay of $0.0005$. We use a smaller batch size and train for fewer steps than Ma et al. [32] to enable an extensive evaluation with repeated experiments. For the same reason, we also train on randomly selected image crops of size $352\times 352$. The only other data augmentation used is random flipping along the vertical axis. We follow Ma et al. [32] and randomly initialize all network weights from $\mathcal{N}(0,10^{-3})$ and all network biases with [math]s. Models are trained on a single NVIDIA TITAN Xp GPU with $12$GB of RAM.

C.2 Description of Results

The results in Figure 6 (Section 4.2) were obtained in the following way:

• •

Ensembling: $33$ models were trained using the same training procedure, the mean and standard deviation was computed based on $32$ ($M=1$), $16$ ($M=2,4,8,16$) or $4$ ($M=32$) sets of randomly drawn models. The same set could not be drawn more than once.

• •

MC-dropout: $16$ models were trained using the same training procedure, based on which the mean and standard deviation was computed.

C.3 Additional Results

Here, we show sparsification plots, sparsification error curves and calibration plots. Examples of sparsification plots are found in Figure 22 for ensembling and Figure 23 for MC-dropout. Condensed sparsification error curves are found in Figure 24 for ensembling and Figure 25 for MC-dropout. Condensed calibration plots are found in Figure 26 for ensembling and Figure 27 for MC-dropout.

Appendix D Street-Scene Semantic Segmentation

In this appendix, further details on the street-scene semantic segmentation experiments (Section 4.3) are provided.

D.1 Training Details

For ensembling, we train all ensemble models for $40\thinspace 000$ steps with SGD + momentum ($0.9$), a batch size of $8$ and weight decay of $0.0005$. The learning rate $\alpha_{t}$ is decayed according to:

[TABLE]

where $T=40\thinspace 000$ and $\alpha_{0}=0.01$ (the initial learning rate). We train on randomly selected image crops of size $512\times 512$. We choose a smaller crop size than Yuan and Wang [51] to enable an extensive evaluation with repeated experiments. The only other data augmentation used is random flipping along the vertical axis and random scaling in the range $[0.5,1.5]$. The ResNet101 backbone is initialized with weights111http://sceneparsing.csail.mit.edu/model/pretrained_resnet/resnet101-imagenet.pth. from a model pretrained on the ImageNet dataset, all other model parameters are randomly initialized using the default initializer in PyTorch. Models are trained on two NVIDIA TITAN Xp GPUs with $12$GB of RAM each. For MC-dropout, models are instead trained for $60\thinspace 000$ steps.

D.2 Description of Results

The results in Figure 8 (Section 4.3) were obtained in the following way:

• •

Ensembling: $26$ models were trained using the same training procedure, the mean and standard deviation was computed based on $8$ sets of randomly drawn models for $M~{}\in~{}\{1,2,4,8,16\}$. The same set could not be drawn more than once.

• •

MC-dropout: $8$ models were trained using the same training procedure, based on which the mean and standard deviation was computed.

D.3 Additional Results

Here, we show sparsification plots, sparsification error curves and reliability diagrams. Examples of sparsification plots are found in Figure 28 for ensembling and Figure 29 for MC-dropout. Condensed sparsification error curves are found in Figure 30 for ensembling and Figure 31 for MC-dropout. Examples of reliability diagrams with histograms are found in Figure 32 for ensembling and Figure 33 for MC-dropout. Condensed reliability diagrams are found in Figure 34 for ensembling and Figure 35 for MC-dropout.