Entropic Out-of-Distribution Detection

TL;DR

This paper introduces IsoMax loss and entropic score as a seamless, efficient alternative to SoftMax for out-of-distribution detection, avoiding accuracy drops and hyperparameter tuning while enhancing detection performance.

Contribution

The paper proposes IsoMax loss and entropic score as novel, practical solutions for OOD detection that require no additional data or hyperparameter tuning, improving efficiency and performance.

Findings

IsoMax loss improves OOD detection accuracy.

Models with IsoMax loss do not suffer accuracy drops.

The approach enables fast, energy-efficient inference.

Abstract

Out-of-distribution (OOD) detection approaches usually present special requirements (e.g., hyperparameter validation, collection of outlier data) and produce side effects (e.g., classification accuracy drop, slower energy-inefficient inferences). We argue that these issues are a consequence of the SoftMax loss anisotropy and disagreement with the maximum entropy principle. Thus, we propose the IsoMax loss and the entropic score. The seamless drop-in replacement of the SoftMax loss by IsoMax loss requires neither additional data collection nor hyperparameter validation. The trained models do not exhibit classification accuracy drop and produce fast energy-efficient inferences. Moreover, our experiments show that training neural networks with IsoMax loss significantly improves their OOD detection performance. The IsoMax loss exhibits state-of-the-art performance under the mentioned conditions (fast energy-efficient inference, no classification accuracy drop, no collection of outlier data, and no hyperparameter validation), which we call the seamless OOD detection task. In future work, current OOD detection methods may replace the SoftMax loss with the IsoMax loss to improve their performance on the commonly studied non-seamless OOD detection problem.

Figures

Tables

Table 1. TABLE I: Classification Accuracy.

Model	Data	Test Accuracy (%) [$\uparrow$]
		SoftMax Loss	IsoMax Loss
	SVHN	96.6	96.6
DenseNet	CIFAR10	95.4	95.2
	CIFAR100	77.5	77.5
	SVHN	96.8	96.8
ResNet	CIFAR10	95.5	95.6
	CIFAR100	77.4	77.3

Table 2. TABLE II: Seamless Out-of-Distribution Detection: No hyperparameter tuning. Fast and energy-efficient inferences. No classification accuracy drop. No outlier/background data.

Model	Data (training)	OOD (unseen)	Seamless OOD Detection: No Classification Accuracy Drop. No Outlier Data.
			Fast and Energy-Efficient Inferences. No Hyperparameter Tuning.
			TNR@TPR951 (%) [$\uparrow$]	AUROC2 (%) [$\uparrow$]	DTACC3 (%) [$\uparrow$]
			SoftMax+MPS4 / SoftMax+ES5 / IsoMax+MPS6 / IsoMax+ES7 (ours)
DenseNet	CIFAR10	SVHN	32.2 / 33.2 / 64.5 / 77.0	86.6 / 86.9 / 94.6 / 96.6	79.9 / 79.9 / 88.1 / 91.6
		TinyImageNet [33]	55.8 / 59.8 / 81.1 / 88.0	93.5 / 94.2 / 96.8 / 97.8	87.6 / 87.8 / 90.8 / 93.2
		LSUN [34]	64.9 / 69.5 / 88.5 / 94.5	95.2 / 95.9 / 97.9 / 98.8	89.9 / 90.0 / 93.1 / 94.9
	CIFAR100	SVHN	20.6 / 24.9 / 27.5 / 23.4	80.1 / 81.9 / 86.3 / 88.6	73.9 / 74.3 / 79.9 / 83.7
		TinyImageNet	19.4 / 23.7 / 42.4 / 49.1	77.0 / 78.8 / 90.2 / 92.6	70.6 / 71.1 / 83.6 / 86.6
		LSUN	18.8 / 24.4 / 48.9 / 63.0	75.9 / 77.9 / 91.3 / 94.7	69.5 / 70.2 / 84.2 / 89.1
	SVHN	CIFAR10	81.5 / 83.7 / 91.6 / 94.1	96.5 / 96.9 / 98.2 / 98.5	91.9 / 92.1 / 94.1 / 95.0
		TinyImageNet	88.2 / 90.0 / 95.3 / 97.0	97.7 / 98.1 / 98.9 / 99.1	93.5 / 93.7 / 95.4 / 96.1
		LSUN	86.4 / 88.4 / 94.7 / 96.8	97.3 / 97.8 / 98.7 / 99.1	92.8 / 93.0 / 95.0 / 95.9
ResNet	CIFAR10	SVHN	43.1 / 44.5 / 81.7 / 83.6	91.7 / 92.0 / 96.8 / 97.1	86.5 / 86.5 / 91.2 / 91.9
		TinyImageNet	46.3 / 48.0 / 66.0 / 70.2	89.8 / 90.0 / 93.9 / 94.6	84.0 / 84.1 / 87.1 / 88.3
		LSUN	51.2 / 53.3 / 76.6 / 82.3	92.2 / 92.6 / 96.2 / 96.9	86.5 / 86.6 / 90.1 / 91.5
	CIFAR100	SVHN	15.9 / 18.0 / 22.5 / 20.2	71.3 / 72.7 / 83.9 / 85.3	66.1 / 66.3 / 77.8 / 79.7
		TinyImageNet	18.5 / 22.4 / 38.9 / 50.6	74.7 / 76.3 / 89.2 / 92.0	68.8 / 69.1 / 82.2 / 85.6
		LSUN	18.4 / 22.4 / 41.4 / 54.8	74.7 / 76.5 / 90.1 / 93.2	69.1 / 69.4 / 83.3 / 87.5
	SVHN	CIFAR10	67.3 / 67.7 / 90.5 / 92.3	89.8 / 89.7 / 97.9 / 98.0	87.0 / 86.9 / 93.7 / 94.1
		TinyImageNet	66.9 / 67.3 / 92.1 / 94.4	89.0 / 89.0 / 98.2 / 98.4	86.7 / 86.6 / 94.3 / 94.8
		LSUN	62.2 / 62.5 / 88.6 / 90.8	86.0 / 85.8 / 97.6 / 97.8	84.2 / 84.1 / 93.4 / 93.6

Table 3. TABLE III: Non-seamless Out-of-Distribution Detection: Unfair comparison of approaches with different special requirements and side effects. No classification accuracy drop. No outlier/background data.

Model	Data (training)	OOD (unseen)	Non-seamless Out-of-Distribution Detection:
			Approaches with Different Special Requirements and Side Effects.
			AUROC (%) [$\uparrow$]	DTACC5 (%) [$\uparrow$]
			ODIN1 / ACET2 / IsoMax+ES3 (ours) / Mahalanobis4
DenseNet	CIFAR10	SVHN	92.8 / NA / 96.6 / 97.6	86.5 / NA / 91.6 / 92.6
		TinyImageNet	97.2 / NA / 97.8 / 98.8	92.1 / NA / 93.2 / 95.0
		LSUN	98.5 / NA / 98.8 / 99.2	94.3 / NA / 94.9 / 96.2
	CIFAR100	SVHN	88.2 / NA / 88.6 / 91.8	80.7 / NA / 83.7 / 84.6
		TinyImageNet	85.3 / NA / 92.6 / 97.0	77.2 / NA / 86.6 / 91.8
		LSUN	85.7 / NA / 94.7 / 97.9	77.3 / NA / 89.1 / 93.8
	SVHN	CIFAR10	91.9 / NA / 98.5 / 98.8	86.6 / NA / 95.0 / 96.3
		TinyImageNet	94.8 / NA / 99.1 / 99.8	90.2 / NA / 96.1 / 98.9
		LSUN	94.1 / NA / 99.1 / 99.9	89.1 / NA / 95.9 / 99.2
ResNet	CIFAR10	SVHN	86.5 / 98.1 / 97.1 / 95.5	77.8 / NA / 91.9 / 89.1
		TinyImageNet	93.9 / 85.9 / 94.6 / 99.0	86.0 / NA / 88.3 / 95.4
		LSUN	93.7 / 85.8 / 96.9 / 99.5	85.8 / NA / 91.5 / 97.2
	CIFAR100	SVHN	72.0 / 91.2 / 85.3 / 84.4	67.7 / NA / 79.7 / 76.5
		TinyImageNet	83.6 / 75.2 / 92.0 / 87.9	75.9 / NA / 85.6 / 84.6
		LSUN	81.9 / 69.8 / 93.2 / 82.3	74.6 / NA / 87.5 / 79.7
	SVHN	CIFAR10	92.1 / 97.3 / 98.0 / 97.6	89.4 / NA / 94.1 / 94.6
		TinyImageNet	92.9 / 97.7 / 98.4 / 99.3	90.1 / NA / 94.8 / 98.8
		LSUN	90.7 / 99.7 / 97.8 / 99.9	88.2 / NA / 93.6 / 99.5

Table 4. TABLE IV: Non-seamless OOD Detection: Inference delays. Presumed computational cost and energy consumption rates.

Model	Data (training)	Hardware (inference)	Non-seamless Out-of-Distribution Detection:
			Inference Delays. Presumed Computational Cost and Energy Consumption Rates.
			SoftMax Loss [35]	IsoMax Loss (ours)	ODIN [11], Mahalanobis [2], Generalized ODIN [9]
			MPS / ES (ms) [$\downarrow$]	MPS / ES (ms) [$\downarrow$]	(ms) [$\downarrow$]
DenseNet	CIFAR10	CPU	18.1 / 19.4	18.0 / 19.2	242.4 ($\approx$ 10x slower)
		GPU	11.6 / 13.0	11.6 / 11.5	39.2 ($\approx$ 4x slower)
	CIFAR100	CPU	18.4 / 19.8	18.4 / 19.3	261.0 ($\approx$ 10x slower)
		GPU	12.9 / 11.4	11.8 / 11.5	39.6 ($\approx$ 4x slower)
	SVHN	CPU	18.1 / 18.6	18.3 / 18.6	241.5 ($\approx$ 10x slower)
		GPU	11.6 / 11.9	11.7 / 11.6	39.6 ($\approx$ 4x slower)
ResNet	CIFAR10	CPU	22.3 / 23.2	23.0 / 23.5	250.4 ($\approx$ 10x slower)
		GPU	4.5 / 3.8	4.2 / 4.1	15.4 ($\approx$ 4x slower)
	CIFAR100	CPU	23.3 / 23.1	23.3 / 23.8	252.6 ($\approx$ 10x slower)
		GPU	4.3 / 3.9	4.3 / 4.2	14.8 ($\approx$ 4x slower)
	SVHN	CPU	23.1 / 23.4	23.4 / 23.3	263.8 ($\approx$ 10x slower)
		GPU	4.2 / 4.0	4.0 / 4.0	15.7 ($\approx$ 4x slower)

Equations

$$\begin{array}{r}{\mathcal{L}}_I({\hat{y}}^{(k)}|\mathbf{x})=-\log \left(\frac{\exp (-d({\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x}),{\mathbf{p}}_{\mathbf{\varphi }}^k))}{\sum _j\exp (-d({\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x}),{\mathbf{p}}_{\mathbf{\varphi }}^j))}\right)\end{array}$$

$$\begin{array}{r}{\mathcal{L}}_I({\hat{y}}^{(k)}|\mathbf{x})=-\log \left(\frac{\exp (-E_s\Vert {\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x})-{\mathbf{p}}_{\mathbf{\varphi }}^k\Vert )}{\sum _j\exp (-E_s\Vert {\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x})-{\mathbf{p}}_{\mathbf{\varphi }}^j\Vert )}\right)\end{array}$$

$$\begin{array}{rl}{p}_I(y^{(i)}|\mathbf{x})& =\frac{\exp (-\Vert {\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x})-{\mathbf{p}}_{\mathbf{\varphi }}^i\Vert )}{\sum _j\exp (-\Vert {\mathbf{f}}_{\mathbf{\theta }}(\mathbf{x})-{\mathbf{p}}_{\mathbf{\varphi }}^j\Vert )}\end{array}$$

Full text

I Introduction

Out-of-distribution (OOD) detection approaches usually use special requirements such as input preprocessing [8, 9], feature extraction combined with metric learning [2], adversarial training [10], hyperparameter validation [11], and collection of additional data [12, 13, 14, 15]. Moreover, current OOD methods commonly show side effects such as classification accuracy drop [16, 9], and slow energy-inefficient inferences [11, 2]. Solutions based on uncertainty (or confidence) estimation (or calibration) present complexity and lead to slow computationally inefficient inferences [17, 18, 19, 20, 21].

We define the seamless OOD detection task, which consists of performing OOD detection under the following restrictions. First, no classification accuracy drop is allowed. Second, the resulting models should produce inferences with the same speed and energy efficiency as those produced by the regularly trained neural networks. Third, no OOD/outlier/additional/extra data may be used. Finally, no hyperparameter validation is required. Improving the performance of neural networks in the seamless OOD detection problem is important from a practical perspective. Additionally, such approaches may be combined in future work with current and novel OOD detection techniques to further improve the performance on the non-seamless OOD detection task.

We argue that the unsatisfactory OOD detection performance of modern neural networks is mainly due to the drawbacks of the SoftMax loss (we follow the “SoftMax loss” expression as defined in [22]). First, the SoftMax loss anisotropy does not incentivize the concentration of high-level representations in the feature space [1, 10], making OOD detection difficult [10] (Fig. 1a). Second, SoftMax loss produces overconfident low-entropy posterior probability distributions [23], which is in disagreement with the maximum entropy principle [24, 25, 26] (Fig. 1b). Therefore, we propose the isotropy maximization loss (IsoMax loss). To fix the SoftMax loss anisotropy, we made IsoMax an isotropic, i.e., exclusively distance-based, loss. To tackle the SoftMax loss overconfidence, we developed the entropy maximization trick, which consists of training with logits multiplied by a high constant that is removed for inference. This technique allows IsoMax loss to produce high-entropy (almost maximum) posterior probability distributions in agreement with the principle of maximum entropy.

We propose to train neural networks replacing the SoftMax loss with the IsoMax loss. The swap of the SoftMax loss with the IsoMax loss requires changes in neither the architecture of the model nor training procedures or parameters. For OOD detection, we use the negative entropy of the neural network output probabilities, which we call the entropic score (ES). Since our solution presents neither special requirements nor side effects, it qualifies as a seamless OOD detection approach as previously defined.

Our contributions are the following. First, we associate the unsatisfactory OOD detection performance of neural networks with the SoftMax loss anisotropy and disagreement with the maximum entropy principle. Second, we propose the IsoMax loss that acts as a SoftMax loss drop-in replacement and may be used as a baseline for building improved OOD detection approaches in future work. We show that the ES produces high performance combined with IsoMax loss. Third, we present the theoretical insight that associates the improved OOD detection performance of the networks trained with IsoMax loss with the principle of maximum entropy. Fourth, we show that our solution produces state-of-the-art performance for the seamless (fast energy-efficient inferences, no classification accuracy drop, no hyperparameter tuning, and no collection of outlier data) OOD detection task. Fifth, despite being unfair since the approaches present different special requirements and side effects, we compare our seamless solution with non-seamless OOD detection methods.

II Related Works

Liang et al. [11] proposed the Out-of-DIstribution detector for Neural networks (ODIN) by combining input preprocessing and temperature calibration. The authors used OOD examples to validate the hyperparameters. Lee at al. [2] proposed the Mahalanobis distance-based approach by adding feature extraction, feature ensembles, and metric learning to input preprocessing. The authors proposed using adversarial examples rather than the OOD samples to tune hyperparameters. Since this procedure produces more realistic estimations, in this work, we only consider validation on adversarial samples for non-seamless OOD detection methods. Hein at al. [10] proposed adversarial confidence enhancing training (ACET), which uses adversarial training.

Hsu et al. [9] proposed to use the in-distribution validation set for hyperparameter tuning. The CIFAR10/100 validation sets were used both for hyperparameter validation and for constructing the OOD detection test sets, which may have produced overestimated results. We believe that the in-distribution validation data used for defining hyperparameters should have been removed from the training set, which would presumably lead to an even stronger classification accuracy drop and, consequently, a decrease in OOD detection performance. The solution used input preprocessing and presented classification accuracy drop of a few percentage points in some cases.

Techapanurak et al. [16] used cosine similarity and a learnable block composed of batch normalization, an exponential, and a linear layer. The trained models presented classification accuracy drop of a few percentage points in some cases. Thus, the authors suggested using two models: one for classification and the other for OOD detection.

Recent approaches [12, 13, 14, 15] increased the OOD performance by training/fine-tuning using outlier data and hyperparameter validation. Liu et al. [15] proposed the energy score. Methods based on uncertainty/confidence estimation/calibration [17, 19, 18, 20, 21] have been proposed to tackle the OOD detection problem.

III Entropic Out-of-Distribution Detection

III-A Isotropy.

To fix the SoftMax loss anisotropy caused by its affine transformation, we forced the logits of the IsoMax loss to depend exclusively on the distances from the high-level features to the class prototypes.

Let $\bm{f}_{\bm{\theta}}(\bm{x})$ represent the high-level feature (embedding) associated with $\bm{x}$, $\bm{p}_{\bm{\phi}}^{j}$ represent the learnable prototype associated with class $j$, and $d()$ represent the nonsquared distance. Additionally, let $\hat{y}^{(k)}$ represent the label of the correct class. Therefore, we construct an isotropic loss by writing:

[TABLE]

Unlike metric learning-based OOD detection approaches, rather than learning a metric from a preexisting feature space, in our solution, we learn a feature space that is from the start consistent with the chosen metric, avoiding the need for feature extraction and metric learning postprocessing phases after the neural network training.

III-B Entropy Maximization.

Isotropy improves the OOD detection performance. However, for further performance gains, we need to circumvent the SoftMax loss propensity to produce low-entropy posterior probability distributions. To achieve high-entropy (almost maximum entropy) distributions in agreement with the maximum entropy principle, we introduce the entropic scale, which consists of a constant scalar factor applied to the logits throughout the training that is nevertheless removed prior to inference. We call this procedure the entropy maximization trick.

The entropic scale is equivalent to the inverse of the temperature of the SoftMax function (we follow the SoftMax function expression as defined in [22]). However, training with a predefined constant entropic scale and then removing it before inference is different from temperature calibration. On the one hand, the temperature of a pretrained model is validated after training and requires access to the OOD or adversarial examples. Furthermore, overoptimistic performance estimation is commonly produced [27]. On the other hand, our approach requires neither hyperparameter validation nor access to the OOD or adversarial data. Rather than be applied to pretrained models, our approach is used to train neural networks.

The presence of the entropic scale during training does not prevent the loss from approaching zero as required. However, when we remove it prior to the inference, the SoftMax function naturally makes the entropy of the output probabilities increase to almost the maximum value possible if we use a high enough entropic score during training. Thus, returning to Equation (1), multiplying the embedding-prototype distances by an entropic scale $E_{s}$, and representing the 2-norm of a vector by $\lVert.\rVert$, we write the definition of the IsoMax loss as:

[TABLE]

By applying the entropy maximization trick, the inference probabilities for the IsoMax loss may be written as follows:

[TABLE]

III-C Prototype Initialization.

We observed that using the Xavier [28] or Kaiming [29] initializations for the prototypes leads to oscillations in performance. Hence, we decided to initialize all prototypes to the zero vector. Weight decay is applied to the prototypes because they are trainable parameters.

III-D Entropic Score.

The entropy has been studied for OOD detection [30]. We show that the output probabilities negative entropy, which we call the entropic score, produces high-performance results when combined with IsoMax loss. Indeed, in such cases, the solution may consider the information provided by all network outputs rather than merely one output. For instance, ODIN and ACET only use the maximum probability.

III-E Implementation Details.

To calculate the losses based on cross-entropy, deep learning libraries usually combine the logarithm and probability into a single computation. However, we experimentally observed that sequentially computing these calculations as standalone operations improves the IsoMax performance. The class prototypes have the same dimension as the neural network last-layer representations. The number of prototypes is equal to the number of classes. The IsoMax loss has fewer parameters than the SoftMax loss because it has no bias to learn.

We observed classification accuracy drop and low/oscillating performance when trying to integrate the entropic scale into the SoftMax loss or with cosine similarity [22, 31, 32]. In such cases, the above strategy, i.e., initialization of the loss weights with the zero vector, cannot be performed. The Mahalanobis distance cannot be used because the covariance matrix is not differentiable. Hence, the nonsquared Euclidean distance is the optimal choice for integration with the entropic scale.

IV Experiments

All datasets, models, and evaluation metrics used the baseline established in [35] that was followed in the major OOD detection papers [11, 2, 10]. We trained from the scratch 100-layer DenseNet-BC [36] (growth rate $k\!=\!12$, 0.8M parameters) and 34-layer ResNets [37] on CIFAR10 [38], CIFAR100 [38] and SVHN [39] using the SoftMax and IsoMax losses.

For both the SoftMax loss and IsoMax loss, we used SGD with the Nesterov moment equal to 0.9, 300 epochs with a batch size of 64, and an initial learning rate of 0.1, with a learning rate decay rate equal to ten applied in epochs 150, 200, and 250. We used a dropout of zero. The weight decay was 0.0001.

We only compared approaches that did not present classification accuracy drop because this facilitates increasing OOD detection performance [40]; moreover, it is particularly undesired from a practical perspective [41]. It is well known that using OOD/outlier/background/additional data improves the OOD detection performance. Therefore, considering that data-based regularization techniques may benefit both SoftMax loss and IsoMax loss, we perform all experiments without outlier exposure [13, 14], background samples [12], or energy-based fine-tuning [15]. The source code is available online111https://github.com/dlmacedo/entropic-out-of-distribution-detection.

V Results and Discussions

V-A IsoMax Loss Properties, Ablation Study, and Entropic Scale Value Definition.

To experimentally show that higher entropic scales lead to higher mean entropy probability distributions and consequently improve the OOD detection performance, we trained DenseNets on SVHN using the SoftMax loss and IsoMax loss with distinct entropic scale values. We used the entropic score and the TNR@TPR95 (true negative rate at 95% true positive rate) to evaluate the OOD detection performance (Fig. 2).

Fig. 2a shows that the SoftMax loss generates posterior distributions with low mean entropy. Fig. 2b illustrates that the unitary entropic scale ($E_{s}\!\!=\!1$) does not increase the mean entropy of the probability distributions. In other words, isotropy alone is not enough to produce low mean entropy probability distributions, and the entropy maximization trick is necessary. Nevertheless, Fig. 2e shows that the simple replacement of anisotropic logits based on the affine transformation by isotropic logits is enough to produce some OOD detection performance gains for all out-of-distribution (out-dist) data, even without the mentioned trick ($E_{s}\!\!=\!1$). Fig. 2c shows that an intermediate entropic scale ($E_{s}\!\!=\!3$) provides medium mean entropy probability distributions with the additional OOD detection performance gains regardless of the out-of-distribution data (Fig. 2e). Fig. 2d illustrates that a high entropic scale ($E_{s}\!\!=\!10$) produces even higher mean entropy probability distributions and the highest OOD detection performance for all out-of-distribution data considered (Fig. 2e). We emphasize that regardless of training with the entropic scale, if it is not removed for inference, the IsoMax loss produces outputs with entropies as low as those produced by the SoftMax loss and high OOD detection performances are no long observed.

Hence, the entropy maximization trick enables the migration from low-entropy distributions (Fig. 2a,b) to high-entropy distributions (Fig. 2d). For a high entropic scale, the IsoMax loss minimizes the cross-entropy while producing high-entropy probability distributions as recommended by the principle of maximum entropy. More importantly, higher entropy posterior probability distributions directly correlate with increased OOD detection performances despite the out-of-distribution data. Fig. 2d shows that an entropic scale $E_{s}\!\!=\!10$ is enough to produce essentially the maximum possible entropy. Therefore, we defined the entropic scale as a constant equal to ten.

V-B Classification Accuracy and OOD Detection Performance Dependence on the Entropic Scale Value.

After defining $E_{s}\!\!=\!10$ for the IsoMax loss based on the previous experiments, we performed additional analyses. Fig. 3 shows that regardless of the combination of dataset and model, the classification accuracy and the mean OOD detection performance are essentially stable for $E_{s}\!\!=\!10$ or higher, as the entropic scale is already high enough to ensure near-maximal entropy. Hence, validation of $E_{s}$ does not produce a considerable performance increase. In fact, this is not even possible because we consider access to OOD or outlier samples to be forbidden. Making $E_{s}$ learnable did not significantly affect the OOD detection performance. Table I shows that the IsoMax loss trained models do not show classification accuracy drop.

V-C Seamless Out-of-Distribution Detection.

To the best of our knowledge, the proposal presented in [35] and our method are the only solutions that qualify as seamless OOD detection approaches. Table II shows that the models trained with the SoftMax loss using the maximum probability as the score (SoftMax+MPS) always present the worst performance results and that replacing the maximum probability score by the entropic score (SoftMax+ES) produces OOD detection performance gains.

The combination of the models trained using IsoMax loss with the entropic score (IsoMax+ES), which is the proposed solution, significantly improves, usually by several percentage points, the OOD detection performance across almost all datasets, models, out-of-distribution data, and metrics.

The entropic score produces high OOD detection performance when the distributions present high entropy (IsoMax+ES). Indeed, both producing high-entropy distributions and the entropic score contribute to improving the OOD detection performance. However, the contribution of producing high-entropy distributions is considerably more important.

The model does not affect the analyses presented. Indeed, the comments shown above are valid for both DenseNet and ResNet models.

V-D Non-seamless Out-of-Distribution Detection.

To tackle non-seamless OOD detection, IsoMax should work as a baseline to be combined with OOD techniques (e.g., outlier exposure, adversarial training, input preprocessing, energy score) rather than competing as a standalone solution. Nevertheless, Table III provides a perspective for how our baseline seamless (standalone) approach compares to non-seamless (composed) solutions.

From a qualitative perspective, ODIN and Mahalanobis use input preprocessing; i.e., to perform OOD detection, each inference requires a first neural network forward pass, a backpropagation, and a second forward pass. They produce slower and less energy-efficient inferences than models trained with IsoMax loss, which are as fast and computationally efficient as the models trained with SoftMax loss. Input preprocessing is indeed a limitation from an economic and environmental perspective [42].

ODIN requires temperature calibration after neural network training, while the Mahalanobis approach requires feature ensemble and metric learning. Unlike pretraining-based solutions, our approach requires no postprocessing after the neural network training. ACET requires adversarial training, which produces slower training and may limit the application of ACET to large images [43].

From a quantitative point of view, Table III shows that IsoMax+ES considerably outperforms ODIN in all evaluated scenarios. Therefore, in addition to avoiding hyperparameter tuning and access to the OOD or adversarial samples, the results show that the entropy maximization trick is much more effective in improving the OOD detection performance than temperature calibration, even when the latter is combined with input preprocessing. Furthermore, IsoMax+ES usually outperforms ACET, in some cases by a large margin. Moreover, in most cases, the Mahalanobis method surpasses IsoMax+ES by less than 2%. In some scenarios, IsoMax+ES outperforms the Mahalanobis method.

Table IV presents the inference delays for the SoftMax loss, IsoMax loss, and competing methods using a CPU and GPU. We observe that neural networks trained using the IsoMax loss produce inferences equally as fast as those produced by networks trained using the SoftMax loss, regardless of whether a CPU or GPU is used for inference.

Additionally, the entropic score is as fast as the usual maximum probability score. Moreover, the methods based on input preprocessing were more than ten times slower on the CPU and approximately four times slower on the GPU. These ratios also presumably apply to the computational cost and energy consumption.

To agree with the maximum entropy principle and achieve high performance, rather than generating calibrated maximum probabilities, IsoMax must produce the lowest possible maximum probabilities.

VI Conclusion

We proposed a seamless OOD detection approach based on logit isotropy and the maximum entropy principle. The proposed IsoMax loss acts as a SoftMax loss drop-in replacement that produces accurate predictions in addition to fast energy- and computation-efficient inferences. No hyperparameter tuning is needed. Hence, no additional procedure other than straightforward neural network training is needed.

OOD detection is performed using the rapid entropic score. Collection of outlier/background data is also not required. To the best of our knowledge, the IsoMax loss does not present any drawbacks compared to the SoftMax loss.

The direct replacement of the SoftMax loss by the IsoMax loss significantly improves the baseline OOD detection performance of neural networks. Therefore, rather than the limitations of the models, the low OOD detection performance of deep networks is due to the SoftMax loss drawbacks, i.e., anisotropy and overconfidence.

In future work, the research community may combine the IsoMax loss with data-based loss regularization techniques [12, 13, 14] to improve the performance. Approaches based on pretrained models [11, 2, 44] or energy-based fine-tuning/score [15] may be applied on IsoMax loss pretrained networks rather than on SoftMax pretrained models.

Thus, rather than competitors, these approaches are actually complementary to IsoMax loss, as they may be combined to achieve even higher overall OOD detection performance. IsoMax loss may replace SoftMax loss as a higher performance baseline for constructing OOD detection solutions.

Another option is to use recent data augmentation techniques [45, 46]. We believe that the simplicity of our solution makes it scalable to large images. Hence, we intend to apply this approach to ImageNet [33]. Finally, since our approach consists of only loss replacement and is based on the general principles of isotropy and maximum entropy, it may be extended to other machine learning methods beyond neural networks.