Online Black-Box Confidence Estimation of Deep Neural Networks

TL;DR

This paper introduces a black-box confidence estimation method called neighborhood confidence (NHC) for deep neural networks, improving real-time reliability assessment in autonomous driving systems without needing access to internal model details.

Contribution

The paper proposes NHC, a novel black-box confidence metric for DNNs that works in real-time and does not require access to gradients or training data, suitable for safety-critical applications.

Findings

NHC performs better or on par with white-box methods in various data shift scenarios.

NHC is effective with limited data regimes, enabling real-time confidence estimation.

Evaluation includes out-of-domain, adversarial, and distribution-shifted data.

Abstract

Autonomous driving (AD) and advanced driver assistance systems (ADAS) increasingly utilize deep neural networks (DNNs) for improved perception or planning. Nevertheless, DNNs are quite brittle when the data distribution during inference deviates from the data distribution during training. This represents a challenge when deploying in partly unknown environments like in the case of ADAS. At the same time, the standard confidence of DNNs remains high even if the classification reliability decreases. This is problematic since following motion control algorithms consider the apparently confident prediction as reliable even though it might be considerably wrong. To reduce this problem real-time capable confidence estimation is required that better aligns with the actual reliability of the DNN classification. Additionally, the need exists for black-box confidence estimation to enable the homogeneous inclusion of externally developed components to an entire system. In this work we explore this use case and introduce the neighborhood confidence (NHC) which estimates the confidence of an arbitrary DNN for classification. The metric can be used for black-box systems since only the top-1 class output is required and does not need access to the gradients, the training dataset or a hold-out validation dataset. Evaluation on different data distributions, including small in-domain distribution shifts, out-of-domain data or adversarial attacks, shows that the NHC performs better or on par with a comparable method for online white-box confidence estimation in low data regimes which is required for real-time capable AD/ADAS.

Full text

I INTRODUCTION

In recent years deep neural networks (DNNs) are increasingly used for advanced driver assistance systems (ADAS) which are deployed in public. Further, DNNs play a key role for full autonomous driving (AD) and enable the most accurate perception, planning, etc. [1]. At the same time, DNNs are quite brittle when the input data does not exactly match the data distribution seen during training. Small domain shifts or the existence of out-of-domain (OOD) data lead to a significant decrease in performance. Such shifts or small corruptions of the data occur naturally and are critical for systems deployed in partly unknown environments like ADAS. Hence, research interest on OOD data started to increase in recent years ([2, 3]).

The decrease in performance on OOD data is further amplified by adversarial attacks [4]. These attacks allow an adversary to fool any machine learning system by generating a small perturbation that is applied on the input data of the system. Initially, such attacks applied the perturbation directly on the input image [5]. More crucial for AD/ADAS are attacks that show the possibility of performing similar attacks in the physical world. Here, patches or markings are placed in a physical scene ([6, 7, 8]) and fool the DNN even after the perturbation is captured by the camera system. It also showed that such physical attacks are possible even if the adversary has only strict black-box access to the system [9], meaning any deployed system could be theoretically attacked. Hence, all described data distribution types are relevant for the safe deployment of AD/ADAS and need to be dealt with.

However, a standard DNN outputs a high confidence in the predictions on these data distributions, meaning a following planning or control algorithm treats the DNN output as a reliable value. For a correct follow-up decision, it is required that the output confidence actually reflects the real reliability of the DNN also under data distribution shifts. Hence, the estimated confidence should align with the true accuracy and data knowledge of the model.

This allows the following algorithms to make an informed decision and not rely on a bad prediction by the DNN. Based on the confidence estimation the following algorithms can for example contact alternative backup systems to override the DNN prediction or engage safety features, like a speed reduction or passing the control to the (safety) driver. Especially, the approach to use different expert models for certain situations is popular [10] and also used by current ADAS systems operating on public roads. However, due to strict timing constraints and limited computational resources it is not possible to run the more complex expert models simultaneously. Therefore, the general DNN must output a meaningful confidence so the controller can decide whether (and which) more complex models are used in the next images of a video stream to generate a specialized understanding of the current scene.

In reality the general DNN might be a combination of different systems from various suppliers, where each is focused on the perception of a certain task, e.g. different systems exist for driving space detection and pedestrian detection. To still use the aforementioned concept of situational expert systems depending on the confidence it is required that the confidence estimation of each supplied system is equally meaningful. One way to ensure this, is by using a unique and reliable confidence estimation that is independent of the individual suppliers. This allows the final manufacturer to assess each system individually.

Therefore, a model agnostic confidence estimation method is required since it is not possible to enforce a certain confidence training method for each supplier, because each has its own pipelines and architectures that cannot be easily adapted to the requirements of each customer. Additionally, we focus on black-box confidence estimation since systems are typically shared by suppliers in a secret fashion where the customer does not have access or knowledge of the individual architecture, components or gradient flows. Instead, only the final output of the supplied system is made available and can be used by the customer for further computations and decisions. Hence, a model agnostic black-box confidence estimation is required to enable the safe usage of systems from different suppliers. An overview of the different confidence estimation categories and our focus area is shown in Fig. 1.

To explore black-box confidence estimation for ADAS we choose a traffic sign recognition (TSR) system as an exemplary ADAS, which is deployed in public by many manufacturers. At the same time, TSR systems are most comparable to work on the confidence estimation of DNNs in the domain of image classification, where most publications are focused on. We choose this use case since it allows to compare current advances from general image classification to our proposed black-box confidence estimation, while still being relevant for ADAS development. This enables us to observe the difference between our black-box method and recent white-box methods.

Our contributions:

• •

We motivate the need for strict black-box confidence estimation that is real-time capable for AD/ADAS

• •

The neighborhood confidence (NHC) is proposed as a method to perform black-box confidence estimation using limited additional samples

• •

A comparison with the most similar online white-box confidence method is performed for small distribution shifts, full OOD data and adversarial attacks

• •

Our findings show that we achieve an improved or similar performance in low data regimes while only using the black-box output which is required for the motivated usage in AD/ADAS

II RELATED WORK

Most publications regarding the confidence estimation of DNNs use methods that change the underlying architecture or training process. This includes the training of multiple models to use ensembles during online inference [11] or using different (probabilistic) layers ([12, 13]) to output a more meaningful probability distribution than the standard softmax layer. These methods are unsuited for our considered setting because they have to be applied in the training process and cannot be used to determine the reliability of a supplied system independent of the concrete supplier and system.

Alternatively, methods exist that do not require the retraining of a model and are added post hoc for inference. Here, one method that also relies on ensembling during inference is Monte-Carlo Dropout [14] which does not require a change to the DNN architecture if dropout [15] is already used during training. In this approach dropout stays active during inference and allows a Bayesian approximation of the confidence. Another method that does not require retraining is temperature scaling ([16, 17]). This improves the calibration of a trained DNN by adding a scaling coefficient to the final logit layer. Furthermore, the combination of a DNN with the k-nearest neighbors algorithm [18] is proposed to estimate the distance of a test data point to the closest train data points. Again, the discussed methods are unsuited for the considered setting because they require an adjustment in the supplied system which a customer typically cannot perform itself.

In our considered setting model agnostic methods are needed that are only applied during inference and do not require any change in the architecture or addition of further components [19]. Here, the most similar and recently proposed method is called attribution-based confidence (ABC) [20] which can be applied to any differentiable model and does not require any changes. It uses the pixel-wise attribution to generate perturbed data points. To determine the attribution gradient-based methods are exploited, meaning the ABC requires white-box access to a model. Hence, using this method in practice requires that supplied systems are shared such that the internal computations are observable. This contrasts with the setting we focus on, which considers secretly sharing a system and thus requires to estimate the confidence of a black-box model.

Additionally, there exists work specific to the separate data distribution types that we consider in this work. Methods are proposed to specifically detect whether a data point is OOD [21] or perturbed by an adversary [22]. However, such methods are not the focus of our work since we are interested in a single system that can provide a meaningful confidence estimation under different data distributions. This is most useful for the scenario motivated in section I because it allows following control algorithms to perform an appropriate action under various different conditions. Using multiple systems to capture each data distribution type is not possible due to strict requirements on the available computational resources and timing constraints.

III NEIGHBORHOOD CONFIDENCE

We first describe the motivation behind our proposed confidence metric. Then, we formulate an algorithm to compute the basic version of the neighborhood confidence. Additionally, we introduce other concepts that improve the performance of the NHC further.

III-A Motivation

The basic idea behind the neighborhood confidence is that the classification reliability of a system is higher when a data point lies in the center of a decision region. At this location the data point is classified as reliable as possible in the associated class, because a small perturbation of the data point does not change the result of the classification. Therefore, the confidence of the system should be highest at such data points to show the highest reliability of the classification.

Following this line of thought, data points near the decision boundary are classified less reliable and should have a lower confidence. Here, a small perturbation is sufficient to push a data point in a different decision region, which leads to a change in the classification without a meaningful change in the data itself. Hence, the confidence of the system should be lower to show that nearly a different class is predicted and the classification is not very reliable.

The described basic concept behind the neighborhood confidence is visualized in Fig. 2 for a two dimensional example with three different classes. Darker colors show a higher value of the classification reliability and consequently an ideal confidence metric should show a similar behavior.

III-B Method

Following the motivation for an ideal confidence metric, a computable black-box metric is needed that reveals how close a decision boundary is to a given data point. Due to the high dimensionality of DNN based systems and input sensors used in reality an ideal metric can only be roughly approximated when considering the strict requirements for AD/ADAS regarding inference time and available computational resources. To perform this approximation, we use a method that tests how robust a classification is under the influence of noise.

First, multiple noise perturbations are added on the input data and then the classification is done by the system on all data points. If the system classifies all perturbed data points as the same class as the unperturbed data point, it shows that the input data point is not near a decision boundary. Otherwise, the influence of the noise would have pushed some perturbed data points in a different decision region and thus a different class. Combining the presented ideas, the final neighborhood confidence to assess the classification reliability of a black-box system is the fraction of perturbed data points that is classified as the same class as the original data point.

Hence, for a generic classification system $f(\cdot)$ the neighborhood confidence $\xi$ for a perturbation strength $\lambda$ can be calculated as:

• •

Obtain raw data point $x\in\mathbb{R}^{D}$

• •

Draw $N$ noise samples $n_{0},\dots,n_{N-1}\in\mathbb{R}^{D}$ from a random distribution

• •

Generate $N$ perturbed data points $x^{\prime}_{0},\dots,x^{\prime}_{N-1}$ with $x^{\prime}_{i}=x+\lambda n_{i}$

• •

Classify all data points $y=[f(x),f(x^{\prime}_{0}),\dots,f(x^{\prime}_{N-1})]$

• •

Calculate the NHC as $\xi=\frac{1}{N}\sum_{i=1}^{N}\{y_{0}==y_{i}\}$

Using this method, the NHC is valued between zero and one where smaller values indicate that a decision boundary is closer. This effect is visualized in Fig. 3 for a simplified two dimensional example. It can be observed that the NHC captures whether a data point is located near the boundary of a decision region or further inside that region. The strength $\lambda$ can be used to adjust the range of the neighborhood sampling to check whether a decision boundary is nearby. If more classes exist and the decision regions are smaller with decision boundaries closer together $\lambda$ can be decreased to still have a meaningful sampling procedure.

The described algorithm allows for an efficient calculation of the NHC, since the classification of all data points can be done in parallel. This is beneficial for the application in environments where strict timing constraints exists, like it is the case for AD/ADAS. If enough computational resources exist the calculation of the NHC can be done without any relevant overhead, by batching all data points $x,x^{\prime}_{0},\dots,x^{\prime}_{N-1}$ together and calculating $y$ with a single forward pass. To exploit this efficiency $N$ must be rather small so that the complete batch fits on the computing device and enough memory is available. Therefore, we study the impact of the number of used noise samples in subsection IV-B in low data regimes.

Furthermore, it is important to note that the presented NHC is model-agnostic and can be calculated without any adaption for black-box systems. This also holds for hard black-box systems [9] where only the top-1 class is output. No information of the classification system $f(\cdot)$ or any internal gradients is required. This allows the application to unknown systems from external suppliers which enables the use case presented in section I.

III-C Enhancements

It is possible to enhance the presented basic version of the NHC in different ways depending on the concrete use case. On one hand, different perturbation strengths $\lambda_{1},\dots,\lambda_{j}$ can be used at the same time instead of only one. This can be useful if the structure of the decision region is unknown or very uneven. It allows to gather more insights into the structure of the surrounding decision boundaries.

Another option is to use the introduced method but specify a concrete reference class for $y_{0}$ instead of using the class $f(x)$ that is predicted by the system on the unperturbed data point. This allows to estimate the distance to the decision boundary of the specified class which is useful if a potential misclassification as a certain class is more severe for some of the classes. For instance, in the case of AD/ADAS one wants to ensure that no pedestrian detection is missed. Hence, the pedestrian class can be chosen as reference class $y_{0}$, which allows to approximate the distance to this class at any time in addition to calculating the normal NHC using $f(x)$ as reference class. If the resulting value for $\xi$ is high in the case that the pedestrian class is used as $y_{0}$ the decision region of the pedestrian class is close and extra care can be taken in subsequent control algorithms.

IV EXPERIMENTS

To evaluate the performance of the neighborhood confidence we perform qualitative experiments on different data distributions types. The performance is compared with the most similar online white-box method ABC [20] which mainly follows our considered setting by not requiring any adjustments or additions in the TSR system. It is interesting to explore whether the usage of the more detailed information in the white-box method achieves a better performance than the proposed black-box method. Since, we are interested in confidence estimation that is real-time capable we only use the gradient from a single backward pass to estimate the attribution required for the ABC, because this results in the least overhead in inference time. Using more computational expensive methods like integrated gradients [23] as explored in [20] would lead to a significant delay in the time required for inference. This goes against our goals and hence we restrict to use a single backward pass.

IV-A Setup

For the DNN based TSR system we choose a standard ResNet-18 architecture [24] that is trained on the German traffic sign recognition benchmark (GTSRB) dataset [25]. Training is performed without additional augmentation and the system achieves a standard accuracy of $\approx$99.3\text{,}\mathrm{\char 37\relax}$$ on the GTSRB final test set.

To evaluate the effect of confidence estimation under a small distribution shift in subsection IV-C we generate synthetic images of the same traffic sign classes that are used in the GTSRB dataset. To this end, we take real images of traffic signs and apply various transformations to simulate different environmental conditions. In this way, we generate $500$ new samples for each of the $43$ classes.

The performance on OOD data is evaluated in subsection IV-D by using the Chinese traffic sign recognition database (TSRD) [26]. Since, we are interested in exploring the performance on full OOD data we drop all images from the TSRD that have a corresponding class in the GTSRB dataset. Thereby, a full OOD dataset of traffic signs is generated where no overlap exists with the classes of the training dataset.

Finally, to generate adversarial attacks in subsection IV-E we use the projected gradient descent (PGD) method [27]. This is a strong and standard choice to evaluate the impact of adversarial attacks and is also used by others ([12, 20]).

IV-B Hyperparameter Study

The limiting factor of the NHC when used for real-time capable AD/ADAS is the number of used noise samples $N$. Hence, we visualize the impact of different choices for $N$ in Fig. 4 for the strength $\lambda=$0.2$$. However, the behavior for other strengths is very similar. It shows, that $N$ mainly represents an option to adjust the granularity of the quantization depending on the use case.

This holds for 4(a) where the standard accuracy of the TSR system is shown in the case that the classification of a data point is disregarded for the calculation of the accuracy when the NHC confidence of this classification is below a threshold. In the case that the confidence threshold equals zero the displayed value is the standard accuracy over all data points, since every classification has a confidence $\xi\geq 0$. Thus, no classification is disregarded for the accuracy calculation. As soon as a confidence threshold greater than zero is used the classifications of data points where the NHC is below this threshold are disregarded and the accuracy is calculated without those classifications. For example, in 4(a) one can observe for $N=$10 that the accuracy is $\approx$95\text{\,}\mathrm{\char 37\relax} when only classifications with a confidence $\xi\geq$0.4$$ are taken into account. In this experiment one expects that the accuracy increases when the classifications with lower confidence are increasingly disregarded.

At the same time, the quantization effect also appears in 4(b). Here, cumulative distribution functions (CDFs) are used to visualize the distribution of the confidence when the classification is performed on data points of unknown classes. For example, one can observe for $N=$2 that $\approx$60\text{\,}\mathrm{\char 37\relax} of all classifications have a confidence $\xi\leq$0.6 or for $N=$5 and $N=$7 that $\approx$40\text{\,}\mathrm{\char 37\relax} of all classifications have a confidence $\xi=$0$$. Because we evaluate on full OOD data it is impossible that the system outputs the correct class and thus a low confidence for each classification is ideal.

From the presented results we take that using $N=$7 achieves a good tradeoff between the quality of the confidence estimation and reduced computational requirements. Hence, in the following we always use $N=$7 noise samples for calculating the NHC. To have a fair comparison we also use the same number of samples for the ABC, since we are interested in a comparison under restricted computational requirements.

Additionally, we analyze the impact of the noise distribution where the samples are drawn from for the NHC. We find that sampling from a Rademacher distribution consistently provides the best results. A possible explanation is that this distribution allows the neighborhood sampling to search in every dimension with the maximum available strength $\lambda$ for decision boundaries. Therefore, for the following results the noise samples are always drawn from a Rademacher distribution.

IV-C In-Domain Distribution Shift

The first comparison of our proposed neighborhood confidence with the attribution-based confidence is shown in Fig. 5. Similar to 4(a), the standard accuracy is shown when classifications of data points are disregarded for the accuracy calculation if the confidence of a classification is below a threshold. We use our generated synthetic dataset to examine the performance under a small distribution shift where similar shifts occur naturally when deploying systems for AD/ADAS in only partly known environments. Also, for the NHC different strengths $\lambda$ are evaluated.

All shown confidence metrics pass the basic sanity check since the accuracy increases when the confidence threshold increases. However, all variants of the NHC reach a higher accuracy for higher confidence thresholds. Also, they show a higher initial increase than the ABC as soon as the confidence threshold is greater than zero. This shows that a significant fraction of the data points that are classified with $\xi=0$ are actually misclassifications. Once these data points are disregarded the accuracy increases notably and keeps climbing monotonously under increased confidence thresholds.

For higher confidence thresholds the NHC also achieves a higher accuracy, meaning less data points with perfect confidence are misclassified than for the ABC. All in all, using the NHC with $\lambda=$0.4$$ leads to the best performance.

IV-D Out-of-Domain Data

In Fig. 6 the second comparison is done by visualizing the distribution of the confidence when classifying data points of unknown classes, similar to 4(b). The OOD dataset is used which consists only of Chinese traffic signs that the TSR system trained on the GTSRB dataset cannot correctly classify. Hence, the ideal alternative is to have a low confidence for every classification and in the best case the confidence is always zero.

A corresponding behavior can be observed for both confidence metrics. For the NHC it shows that the strength impacts the overall confidence level. Higher strengths lead to a decreased confidence on most data points which is intuitive. The previously best version with $\lambda=$0.4 performs on par with the ABC and is only slightly outperformed using $\lambda=$0.5.

IV-E Adversarial Attacks

Lastly, we compare the impact of an adversary on the ABC and NHC in Fig. 7. The synthetic dataset is used again, and the confidence is evaluated while increasingly severe PGD attacks [27] are performed on the TSR system. Here, $\epsilon$ denotes the severity of the adversary in terms of $\ell_{\infty}$ norm and $\epsilon=0$ means no adversary is present. Therefore, this is equivalent to the setting of standard classification.

The first observation is that all confidence metrics successfully decrease the confidence as soon as the adversary is introduced. However, for the lowest strength $\lambda=$0.3$$ the confidence begins to significantly increase again once the severity of the adversary is further increased. In this case, a more severe adversary can reduce the impact of the NHC because the increased severity of the adversarial perturbation pushes the data points further into the decision region of the target class of the adversary. Using higher strengths for the neighborhood sampling prevents this effect, since the check for decision boundaries is performed at a greater distance.

Another interesting point for observation is the value of the confidence for $\epsilon=0$. Here, no adversary exists meaning the resulting values are the mean confidence on the unperturbed synthetic dataset. Intuitively, the mean confidence decreases when $\lambda$ is increased for the NHC. However, the value is also rather low for the ABC. This means that some variants assign a low confidence to most of the classifications. The general confidence level is sometimes low which harms the ability to correctly distinguish between benign and harmful data points when the difference to the confidence level under influence of an adversary is too low. In section V we further elaborate on this behavior and the origin.

Similar to Fig. 5 the NHC with $\lambda=0.4$ achieves the best results because the tradeoff is optimized when considering the standard mean confidence and the meaningful confidence decrease under the influence of the adversary. This version can detect if a significant change in the distribution of the data points exists and reflects this change in the confidence. It performs best (or close to best) on all experiments showing that a single optimal strength value can be selected which allows the efficient usage of the NHC in real applications.

V DISCUSSION

Our experiments show that for the considered low data regimes the additionally available information used in a strong gradient-based white-box method cannot be exploited and provide no benefit over the neighborhood confidence. Instead, drawing from a Rademacher distribution provides better confidence estimates in most considered cases. This is promising for the application in AD/ADAS since less complex methods are needed to comply with the strict timing requirements.

Our results in subsection IV-E show that the general confidence level is rather low for some evaluated variants also on unperturbed data. The use of the synthetic dataset represents a small in-domain distribution shift that causes the data points to spread more over a decision region and lie closer to a decision boundary. In some cases the data points also lie in a different decision region since the standard accuracy drops from $\approx$99.3\text{,}\mathrm{\char 37\relax} on the original GTSRB test dataset to $\approx$92.9\text{\,}\mathrm{\char 37\relax} on our synthetic dataset (see 4(a) or Fig. 5). The reliability of the classification is reduced which is reflected in all evaluated confidence metrics. However, for some variants the confidence level on benign data points under this distribution shift is rather low and one might want to increase the confidence gap to actually perturbed and harmful data points that are important to distinguish. To accomplish this the integration of concepts for calibration [16] in online confidence metrics seems promising. It is interesting to explore the calibration of online metrics for confidence estimation depending on the current data distribution observed in past data points during inference.

Finally, we like to point out that it is in principle possible to combine the NHC with training methods for an improved confidence estimation. One could explore whether the use of augmentation methods during training, like AugMix [28], has an impact on the confidence estimation. In our preliminary experiments strong augmentation during training led to larger and more robust decision regions. This mainly improves the behavior of all evaluated confidence metrics on unperturbed data by increasing the average confidence, while keeping the strong performance on other distribution types. Similarly, the interaction of the NHC with adversarial training [27] merits a detailed investigation because adversarial training leads to increased and homogeneous decision regions around the training data samples.

VI CONCLUSION

We introduce the neighborhood confidence for online black-box confidence estimation of DNNs motivated by searching the neighborhood of a data point for different decision boundaries. No information of the DNN is required and only the top-1 class output is used, which is the minimum possible output of a DNN. This allows to use the NHC to assess the classification reliability of externally supplied components. The performance of the NHC is evaluated for different data distribution types deviating from the training data distribution allowing only strictly limited additional samples for inference, as required for AD/ADAS. In this low data regime, the NHC performs better or similar to the most comparable method from the literature, even though this attribution-based confidence requires white-box access to the DNN.