Multi-Method Ensemble for Out-of-Distribution Detection

TL;DR

This paper introduces a Multi-Method Ensemble (MME) score that combines multiple feature truncation and scoring techniques to improve out-of-distribution detection in neural networks, demonstrating significant performance gains across various benchmarks.

Contribution

The paper proposes a novel ensemble approach that unifies and combines multiple state-of-the-art OOD detection methods, enhancing robustness and accuracy.

Findings

MME outperforms recent methods on large-scale benchmarks.

Achieves an average FPR95 of 27.57% on ImageNet-1K.

Combining multiple techniques improves OOD detection robustness.

Abstract

Detecting out-of-distribution (OOD) samples is essential for neural networks operating in open-world settings, particularly in safety-critical applications. Existing methods have improved OOD detection by leveraging two main techniques: feature truncation, which increases the separation between in-distribution (ID) and OOD samples, and scoring functions, which assign scores to distinguish between ID and OOD data. However, most approaches either focus on a single family of techniques or evaluate their effectiveness on a specific type of OOD dataset, overlooking the potential of combining multiple existing solutions. Motivated by this observation, we theoretically and empirically demonstrate that state-of-the-art feature truncation and scoring functions can be effectively combined. Moreover, we show that aggregating multiple scoring functions enhances robustness against various types of OOD samples. Based on these insights, we propose the Multi-Method Ensemble (MME) score, which unifies state-of-the-art OOD detectors into a single, more effective scoring function. Extensive experiments on both large-scale and small-scale benchmarks, covering near-OOD and far-OOD scenarios, show that MME significantly outperforms recent state-of-the-art methods across all benchmarks. Notably, using the BiT model, our method achieves an average FPR95 of 27.57% on the challenging ImageNet-1K benchmark, improving performance by 6% over the best existing baseline.

Tables

Table 1. Table 1: Comparison between different methods in OOD detection on small-scale CIFAR benchmarks using a DenseNet model. Values are average percentages over six OOD datasets.

ID Dataset	Metrics	Methods
		MSP	MLS	ViM	ASH-S	GEN	SCALE	VRA	fDBD	MME
CIFAR-10	AUROC$\uparrow$	92.17	93.48	96.58	94.93	94.88	93.90	94.92	96.14	96.92
	FPR95$\downarrow$	47.11	29.63	16.48	23.36	30.01	28.59	27.14	20.52	15.60
CIFAR-100	AUROC$\uparrow$	75.62	81.32	90.95	90.44	81.48	88.52	87.08	90.73	92.48
	FPR95$\downarrow$	77.52	68.64	38.36	40.95	68.54	47.59	61.63	44.61	28.96

Table 2. Table 2: OOD detection for MME and state-of-the-art methods. The ID dataset is ImageNet-1K and the OOD datasets are iNaturalist, Places, SUN and Textures. Results are reported using AUROC and FPR95 metrics (in percentage). A pre-trained BiT and a ViT model are tested. The best method is highlighted in bold, and the 2nd and 3rd best are underlined.

Method	iNaturalist		Places		SUN		Textures		Average
	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$
Backbone : BiT
MSP (ICLR’17)	87.90	64.58	79.27	79.00	81.82	71.82	79.76	77.17	82.19	73.14
MLS (ICML’22)	86.78	70.51	82.58	74.36	86.74	63.49	81.65	73.60	84.44	70.49
Energy (NeurIPS’20)	84.52	74.96	82.56	73.21	87.17	60.81	81.10	73.91	83.84	70.72
ReAct (NeurIPS’21)	91.50	48.65	87.96	54.67	92.33	38.99	90.65	50.14	90.61	48.11
Maha (NeurIPS’18)	85.80	64.69	73.06	82.35	78.10	73.38	97.33	13.95	83.57	58.59
KL (ICML’22)	90.84	38.19	73.17	79.10	76.43	75.88	83.22	55.16	80.91	62.08
ViM (CVPR’22)	89.38	55.09	83.33	67.94	87.85	57.79	98.92	4.63	89.87	46.36
DICE (ECCV’22)	91.44	43.64	88.99	49.20	92.52	35.54	83.67	55.10	89.16	45.87
ASH-S (ICLR’23)	93.87	36.85	88.49	49.17	91.77	34.69	97.62	12.03	92.94	33.19
SHE (ICLR’23)	77.65	72.25	80.31	71.63	84.65	56.68	84.58	48.84	81.80	62.35
GEN (CVPR’23)	88.67	68.32	80.19	79.66	84.17	72.23	81.48	77.91	83.63	74.53
PCA (ICCV’23)	50.37	99.77	42.19	99.80	50.15	99.25	69.69	85.35	53.1	96.04
NNGuide (ICCV’23)	88.48	52.83	88.17	50.76	92.92	33.58	94.94	21.51	91.11	40.60
SCALE (ICLR’24)	93.77	39.37	88.96	50.10	92.39	35.28	96.11	18.99	92.81	35.94
VRA (NeurIPS’24)	91.50	48.65	87.96	54.67	92.33	38.99	90.65	50.14	90.61	48.11
fDBD (ICML’24)	91.71	51.42	84.20	67.45	88.47	56.03	95.45	24.38	89.96	49.82
MME (Ours)	95.53	26.91	89.73	45.03	92.70	30.64	98.45	7.71	94.10	27.57
Backbone : ViT
MSP (ICLR’17)	96.13	19.15	85.09	59.97	86.13	57.00	87.13	48.45	88.62	46.14
MLS (ICML’22)	98.57	6.53	89.24	46.65	91.50	38.97	93.05	30.27	93.09	30.61
Energy (NeurIPS’20)	98.66	6.04	89.30	45.27	91.75	37.09	93.42	28.12	93.28	29.13
ReAct (NeurIPS’21)	99.01	4.19	89.11	47.37	91.54	39.04	93.38	28.37	93.26	29.74
Maha (NeurIPS’18)	99.63	2.15	86.29	60.04	89.11	51.29	94.21	25.27	92.31	34.69
KL (ICML’22)	96.62	16.45	84.34	63.50	85.91	59.41	88.26	45.99	88.78	46.34
ViM (CVPR’22)	99.41	2.56	88.47	50.44	91.53	39.48	95.31	20.14	93.68	28.15
DICE (ECCV’22)	65.49	91.69	61.56	93.88	64.75	88.51	75.49	74.26	66.82	87.09
ASH-S (ICLR’23)	61.96	95.34	61.23	95.68	65.52	92.90	56.79	91.12	61.38	93.76
SHE (ICLR’23)	90.69	43.50	85.63	59.25	87.97	51.93	92.18	32.31	89.12	46.75
GEN (CVPR’23)	98.63	5.83	89.22	49.61	91.00	43.36	92.35	34.05	92.80	33.21
PCA (ICCV’23)	95.36	27.37	76.36	79.58	80.92	72.92	90.20	43.59	85.71	55.86
NNGuide (ICCV’23)	87.25	52.54	80.85	62.61	86.04	51.16	91.97	32.66	88.03	46.05
SCALE (ICLR’24)	98.88	5.00	90.14	42.89	92.57	34.28	93.39	28.51	93.75	27.67
VRA (NeurIPS’24)	99.52	1.89	90.75	42.98	92.99	34.49	93.37	29.19	94.16	27.14
fDBD (ICML’24)	98.39	7.15	89.07	48.68	91.13	41.35	93.10	29.81	92.92	31.75
MME (Ours)	99.48	2.16	89.66	46.71	92.49	35.92	95.48	19.77	94.28	26.14

Table 3. Table 3: Ablation study on impact of NME + \text{NME}^{+} and CO + \text{CO}^{+} . Results on ImageNet-1K with ViT model. The best method is highlighted in bold, and the 2nd and 3rd best are underlined.

Method	iNaturalist		Places		SUN		Textures		Average
	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$
MSP (ICLR’17)	96.13	19.15	85.09	59.97	86.13	57.00	87.13	48.45	88.62	46.14
MLS (ICML’22)	98.57	6.53	89.24	46.65	91.50	38.97	93.05	30.27	93.09	30.61
Energy (NeurIPS’20)	98.66	6.04	89.30	45.27	91.75	37.09	93.42	28.12	93.28	29.13
ReAct (NeurIPS’21)	99.01	4.19	89.11	47.37	91.54	39.04	93.38	28.37	93.26	29.74
Maha (NeurIPS’18)	99.63	2.15	86.29	60.04	89.11	51.29	94.21	25.27	92.31	34.69
KL (ICML’22)	96.62	16.45	84.34	63.50	85.91	59.41	88.26	45.99	88.78	46.34
ViM (CVPR’22)	99.41	2.56	88.47	50.44	91.53	39.48	95.31	20.14	93.68	28.15
DICE (ECCV’22)	65.49	91.69	61.56	93.88	64.75	88.51	75.49	74.26	66.82	87.09
ASH-S (ICLR’23)	61.96	95.34	61.23	95.68	65.52	92.90	56.79	91.12	61.38	93.76
SHE (ICLR’23)	90.69	43.50	85.63	59.25	87.97	51.93	92.18	32.31	89.12	46.75
GEN (CVPR’23)	98.63	5.83	89.22	49.61	91.00	43.36	92.35	34.05	92.80	33.21
PCA (ICCV’23)	95.36	27.37	76.36	79.58	80.92	72.92	90.20	43.59	85.71	55.86
NNGuide (ICCV’23)	87.25	52.54	80.85	62.61	86.04	51.16	91.97	32.66	88.03	46.05
SCALE (ICLR’24)	98.88	5.00	90.14	42.89	92.57	34.28	93.39	28.51	93.75	27.67
VRA (NeurIPS’24)	99.52	1.89	90.75	42.98	92.99	34.49	93.37	29.19	94.16	27.14
fDBD (ICML’24)	98.39	7.15	89.07	48.68	91.13	41.35	93.10	29.81	92.92	31.75
${\text{NME}}^{+}$	98.13	7.92	87.73	52.09	89.94	45.49	93.47	28.51	92.32	33.50
${\text{NME}}^{+}$ + ${\text{CO}}^{+}$	98.59	6.22	88.18	50.98	90.22	44.57	93.78	27.19	92.69	32.24
MME	99.48	2.16	89.66	46.71	92.49	35.92	95.48	19.77	94.28	26.14

Table 4. Table 4: Covariance between six state-of-the-art OOD detectors on the ID dataset ImageNet-1K / averaged across four OOD datasets (iNaturalist, Places, SUN, and Textures), using a ViT model.

ID / OOD	MLS	Energy	ViM	GEN	NNGuide	fDBD
MLS		4.44 / 3.88	6.12 / 5.26	16.03 / 15.47	9.00 / 8.21	109.76 / 105.19
Energy	4.44 / 3.88		5.62 / 4.16	14.48 / 11.80	8.29 / 6.48	99.09 / 78.81
ViM	6.12 / 5.26	5.62 / 4.16		20.18 / 15.81	11.15 / 8.75	145.68 / 110.26
GEN	16.03 / 15.47	14.48 / 11.80	20.18 / 15.81		26.72 / 23.39	370.00 / 319.26
NNGuide	9.00 / 8.21	8.29 / 6.48	11.15 / 8.75	26.72 / 23.39		199.56 / 162.68
fDBD	109.76 / 105.19	99.09 / 78.81	145.68 / 110.26	370.00 / 319.26	199.56 / 162.68

Table 5. Table 5: Ablation study on the variation of feature truncation methods impact each component of MME. Results on ImageNet-1K with ViT model.

Method	Feature truncation	iNaturalist		Places		SUN		Textures		Average
		AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$
Energy	Base	99.40	2.48	88.83	48.91	91.70	38.82	95.35	20.02	93.82	27.56
	VRA	99.63	1.57	89.34	49.61	92.12	38.46	95.18	20.81	94.07	27.61
ViM	Base	99.51	2.04	89.55	46.94	92.46	35.82	95.51	19.65	94.26	26.11
	SCALE	99.33	2.68	88.43	51.41	91.29	41.59	95.25	20.85	93.58	29.13
fDBD	Base	99.45	2.26	89.61	46.81	92.45	35.94	95.49	19.79	94.25	26.20
	SCALE	99.45	2.26	89.61	46.80	92.45	35.92	95.49	19.77	94.25	26.19
PCA	Base	99.46	2.25	89.61	46.74	92.44	35.98	95.47	19.83	94.24	26.20
	SCALE	99.46	2.25	89.61	46.74	92.44	35.98	95.47	19.83	94.24	26.20
${\text{NME}}^{+}$	SCALE	99.48	2.17	89.66	46.72	92.49	35.93	95.48	19.78	94.28	26.15
	VRA	99.52	2.05	89.73	46.77	92.54	35.84	95.45	20.04	94.31	26.17
Baseline		99.48	2.16	89.66	46.71	92.49	35.92	95.48	19.77	94.28	26.14

Table 6. Table 6: Ablation study on adding more scoring function to MME. Results on ImageNet-1K with ViT model.

Scoring function	iNaturalist		Places		SUN		Textures		Average
	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$	AUROC$\uparrow$	FPR95$\downarrow$
GEN	99.15	3.77	89.84	47.70	91.89	39.39	93.79	28.24	93.67	29.77
NNGuide	99.43	2.36	89.49	47.15	92.41	36.10	95.51	19.50	94.21	26.28
SHE	99.43	2.31	89.71	46.72	92.47	35.92	95.48	19.63	94.27	26.15
$\varnothing$	99.48	2.16	89.66	46.71	92.49	35.92	95.48	19.77	94.28	26.14

Equations

$$\mathbf{x}\in \{\begin{array}{ll}\text{ID}& \text{if}S(\mathbf{x})\ge \tau \\ \text{OOD}& \text{if}S(\mathbf{x})<\tau \end{array}$$

$${\mathbb{E}}_{\text{in}}[S(\mathbf{x})]\ge {\mathbb{E}}_{\text{out}}[S(\mathbf{x})]$$

$${\mathbb{E}}_{\text{in}}[g(z)]-{\mathbb{E}}_{\text{out}}[g(z)]\ge {\mathbb{E}}_{\text{in}}[z]-{\mathbb{E}}_{\text{out}}[z]$$

$$\text{CO}(\mathbf{x})=\{\begin{array}{ll}1,& \text{if }{c}_{\text{NME+}}(\mathbf{x})=c_{\text{MLS}}(\mathbf{x}),\\ 0,& \text{otherwise.}\end{array}.$$

$${\text{CO}}^{+}(\mathbf{x})=\{\begin{array}{ll}\lambda ,& \text{if }{c}_{\text{NME+}}(\mathbf{x})=c_{\text{MLS}}(\mathbf{x}),\\ 1,& \text{otherwise.}\end{array}.$$

$$\mathbb{V}(S_1\cdot \dots \cdot S_n)\ge \max \left(\mathbb{V}(S_1),\dots ,\mathbb{V}(S_n)\right).$$

$$\begin{array}{rl}\mathbb{V}(S_1\cdot S_2)& \ge {\mathbb{E}}_{\text{in}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{in}}(S_2(\mathbf{x}))-{\mathbb{E}}_{\text{out}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))\\ & \ge \mathbb{V}(S_1)\end{array}$$

$${\mathbb{E}}_{\text{in}}[S(g(z))]-{\mathbb{E}}_{\text{out}}[S(g(z))]\ge {\mathbb{E}}_{\text{in}}[S(z)]-{\mathbb{E}}_{\text{out}}[S(z)]$$

$$\text{MME}(\mathbf{x})=\exp (\text{SCALE}(\mathbf{x})-\text{ViM}(\text{VRA}(\mathbf{x})))\cdot \text{fDBD}(\text{VRA}(\mathbf{x}))\cdot \text{PCA}(\text{VRA}(\mathbf{x}))\cdot {\text{CO}}^{+}(\mathbf{x})\cdot {\text{NME}}^{+}(\mathbf{x})$$

$$\begin{array}{rl}\mathbb{V}(S_1\cdot S_2)& ={\mathbb{E}}_{\text{in}}[S_1\cdot S_2(\mathbf{x})]-{\mathbb{E}}_{\text{out}}[S_1\cdot S_2(\mathbf{x})]\\ & ={\mathbb{E}}_{\text{in}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{in}}(S_2(\mathbf{x}))+{\text{Cov}}_{\text{in}}(S_1,S_2)-{\mathbb{E}}_{\text{out}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))-{\text{Cov}}_{\text{out}}(S_1,S_2)\end{array}$$

$$\begin{array}{rl}\mathbb{V}(S_1\cdot S_2)\ge & {\mathbb{E}}_{\text{in}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{in}}(S_2(\mathbf{x}))-{\mathbb{E}}_{\text{out}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))\end{array}$$

$$\begin{array}{rl}\mathbb{V}(S_1\cdot S_2)& \ge {\mathbb{E}}_{\text{in}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))-{\mathbb{E}}_{\text{out}}(S_1(\mathbf{x})){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))\\ & \ge ({\mathbb{E}}_{\text{in}}(S_1(\mathbf{x}))-{\mathbb{E}}_{\text{out}}(S_1(\mathbf{x}))){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))\\ & \ge \mathbb{V}(S_1){\mathbb{E}}_{\text{out}}(S_2(\mathbf{x}))\end{array}$$

$$\mathbb{V}(S_1\cdot S_2)\ge \mathbb{V}(S_1)$$

$${\mu }_y=\frac{1}{|{\mathcal{P}}_y|}\sum _{x\in {\mathcal{P}}_y}\varphi (x)$$

$$\text{NME}(\mathbf{x})=\arg \underset{y\in \mathcal{C}}{\min }{\Vert \varphi (\mathbf{x})-{\mu }_y\Vert }_2$$

$${\text{NME}}^{+}(\mathbf{x})=\max (\frac{1}{\text{softmax}(\frac{\text{NME}(\mathbf{x})}{T})})$$

Full text

1 Introduction

With the growing deployment of neural networks in real-world applications, they may encounter samples outside their trained class set. The unpredictable nature of such out-of-distribution (OOD) instances can prevent the model from making confident and reliable predictions [Yang et al.(2022)Yang, Wang, Zou, Zhou, Ding, Peng, Wang, Chen, Li, Sun, et al.]. Therefore, OOD detection is essential for ensuring the safety of machine learning systems, particularly in high-stakes applications such as autonomous driving [Chitta et al.(2021)Chitta, Prakash, and Geiger, Vojir et al.(2021)Vojir, Šipka, Aljundi, Chumerin, Reino, and Matas], medical imaging [Schlegl et al.(2019)Schlegl, Seeböck, Waldstein, Langs, and Schmidt-Erfurth, Nair et al.(2020)Nair, Precup, Arnold, and Arbel] and biosynthesis [Vamathevan et al.(2019)Vamathevan, Clark, Czodrowski, Dunham, Ferran, Lee, Li, Madabhushi, Shah, Spitzer, et al.].

The standard approach to OOD detection involves deriving a score function from a trained neural network, where in-distribution (ID) samples receive relatively higher scores than OOD samples. Researchers have developed scoring functions using either feature representations [Lee et al.(2018)Lee, Lee, Lee, and Shin, Sun et al.(2022)Sun, Ming, Zhu, and Li] or logit values [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song, Liu et al.(2020)Liu, Wang, Owens, and Li], aiming to identify properties that naturally distinguish ID and OOD samples. More recent work [Wang et al.(2022)Wang, Li, Feng, and Zhang, Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang, Park et al.(2023)Park, Jung, and Teoh] has even combined both types of information sources to better handle the wide variety of OOD instances. However, while combining multiple inputs has been shown to be beneficial, these methods sometimes achieve suboptimal performance in specific applications [Zhang et al.(2023)Zhang, Yang, Wang, Wang, Lin, Zhang, Sun, Du, Li, Liu, et al.], even compared to simpler scoring functions. In contrast, feature truncation methods [Sun et al.(2021)Sun, Guo, and Li, Sun and Li(2022)] aim to increase the separation between ID and OOD samples by modifying the network’s propagated signals or weights. Despite their simplicity, feature truncation techniques have been shown to effectively reduce the overlap between ID and OOD samples at both the feature and logit levels. However, these methods cannot be used independently and are typically combined with simple scoring functions, such as Energy [Liu et al.(2020)Liu, Wang, Owens, and Li] or MLS [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song, Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman], for OOD detection. Therefore, even state-of-the-art feature truncation methods [Xu et al.(2023a)Xu, Chen, Franchi, and Yao, Xu et al.(2023b)Xu, Lian, Liu, and Tao], when paired with such scoring functions, often fail to outperform more advanced scoring methods due to the inherent simplicity of their approach.

In this paper, we empirically and theoretically demonstrate that combining multiple scoring functions with feature truncation methods improves OOD detection by better handling the diverse range of OOD examples encountered in real-world scenarios. Building on this ensemble approach, we propose a Multi-Method Ensemble (MME) score, which integrates recent scoring functions with feature truncation techniques. Specifically, the scoring process begins by extracting features, which are then refined using SCALE [Xu et al.(2023a)Xu, Chen, Franchi, and Yao] and VRA [Xu et al.(2023b)Xu, Lian, Liu, and Tao] feature truncation to obtain sparse and clipped representations. These processed features are subsequently used by state-of-the-art scoring functions, including fDBD [Liu and Qin(2023)], PCA [Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang], ViM [Wang et al.(2022)Wang, Li, Feng, and Zhang], and our proposed $\text{NME}^{+}$ method, to generate multiple scalar scores. Finally, these individual scores are combined to produce a final OOD score. The design of MME intuitively ensures that even if one method fails to detect an OOD sample, another method is likely to do so, thereby improving overall detection performance, as illustrated in Figure˜1.

To verify MME effectiveness, we perform experiments on several benchmark datasets and models, including CIFAR-10, CIFAR-100 [Krizhevsky et al.(2009)Krizhevsky, Hinton, et al.], and the more challenging ImageNet-1K [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.]. The results show that our method outperforms current post-hoc strategies, achieving state-of-the-art performance. The key contributions of this paper are summarized as follows:

• •

Theory: We theoretically demonstrate that combining multiple scoring functions improves OOD detection. Additionally, we empirically show that scoring functions generally perform better when fused with feature truncation methods.

• •

Methodology: We propose an effective post-hoc strategy called MME, which combines multiple state-of-the-art OOD detection methods to better handle the wide variety of OOD samples.

• •

Performance: We evaluate MME on popular benchmarks, including both large-scale and small-scale scenarios, as well as near-OOD and far-OOD cases. Our results show that MME consistently outperforms existing post-hoc methods for OOD detection.

2 Related Work

OOD detection research focuses on two approaches: feature truncation [Sun et al.(2021)Sun, Guo, and Li] and creating a scalar score function to distinguish between OOD and ID samples [Hendrycks and Gimpel(2016)].

Feature truncation methods [Sun et al.(2021)Sun, Guo, and Li, Sun and Li(2022)] aim to better separate ID and OOD samples by altering network signals or weights. ReAct [Sun et al.(2021)Sun, Guo, and Li] clips high activation values to disrupt OOD signals while preserving ID ones. VRA [Xu et al.(2023b)Xu, Lian, Liu, and Tao] goes further by also suppressing low and amplifying mid-range activations. ASH [Djurisic et al.(2022)Djurisic, Bozanic, Ashok, and Liu] prunes and scales activations, while SCALE [Xu et al.(2023a)Xu, Chen, Franchi, and Yao] avoids pruning to reduce ID-OOD overlap. However, feature truncation methods cannot be applied by themselves for detecting OOD and must be combined with a score function.

Scalar score functions for OOD detection are generally divided into classifier-based and distance-based approaches. Classifier-based methods, or confidence scores, use neural network outputs to derive scores. MSP [Hendrycks and Gimpel(2016)] uses maximum softmax probability for ID detection, while MLS [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song, Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman] uses logits to preserve more information. Energy score [Liu et al.(2020)Liu, Wang, Owens, and Li] quantifies prediction uncertainty by mapping model outputs to an energy value. Distance-based methods determine OOD samples by measuring distances in feature space. The Mahalanobis detector [Lee et al.(2018)Lee, Lee, Lee, and Shin] uses class-wise mean distances and shared covariances. In contrast, fDBD [Liu and Qin(2023)] focuses on distances from decision boundaries, showing ID features lie further away.

Classifier-based detectors effectively identify near-OOD instances near decision boundaries but exhibit overconfidence in far-OOD regions [Hein et al.(2019)Hein, Andriushchenko, and Bitterwolf]. In contrast, distance-based methods perform well in far-OOD detection but struggle with near-OOD samples close to ID classes. Recent methods [Wang et al.(2022)Wang, Li, Feng, and Zhang, Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang, Park et al.(2023)Park, Jung, and Teoh] address these limitations by combining classifier- and distance-based cues. ViM [Wang et al.(2022)Wang, Li, Feng, and Zhang] combines virtual logits from feature residuals with an energy score, while NNGuide [Park et al.(2023)Park, Jung, and Teoh] improves logits-based confidence using KNN guidance in feature space.

3 Methods

3.1 Preliminaries

To ensure consistency with previous work [Sun et al.(2021)Sun, Guo, and Li, Xu et al.(2023b)Xu, Lian, Liu, and Tao, Park et al.(2023)Park, Jung, and Teoh], we define the OOD detection task as follows: let $\mathcal{X}$ be the input space and $\mathcal{Y}$ be a label space containing $k$ distinct categories. Consider a neural network $f:\mathcal{X}\to\mathbb{R}^{k}$ trained on samples drawn from the in-distribution data. The goal of OOD detection is to design a scoring function, $S$, that determines whether a test input $\mathbf{x}\in\mathcal{X}$ belongs to the ID or OOD space based on its score $S(\mathbf{x})$ and a threshold $\tau$.

[TABLE]

By convention, scoring functions are designed to produce high values for in-distribution data and low values for out-of-distribution data. Therefore, an effective scoring function should satisfy the following

[TABLE]

Let define $\mathbb{V}(S)=\mathbb{E}_{\text{in}}[S(\mathbf{x})]-\mathbb{E}_{\text{out}}[S(\mathbf{x})]$, by definition, OOD detection objective is to find $S$ that maximizes $\mathbb{V}(S)$.

Feature truncation aims to increase the separation between ID and OOD samples by adjusting the network’s propagated signals or weights. Let $g$ be a feature truncation method and $z$ a feature vector, where $z=f(\mathbf{x})$. We adopt the terminology introduced by [Sun et al.(2021)Sun, Guo, and Li, Xu et al.(2023b)Xu, Lian, Liu, and Tao].

[TABLE]

For convenience, since scoring functions typically use feature vectors or logits to compute a confidence score, we adopt the notation $S(\mathbf{x})$ or $S(z)$, depending on the context.

In the Section˜3.3, we will demonstrate that combining two scoring functions can improve OOD detection performance. To achieve this, we make the following assumption:

Assumption 1**.**

Let $S_{1}$ and $S_{2}$ be two scoring functions and $\text{Cov}(S_{1},S_{2})$ the covariance of $S_{1}$ and $S_{2}$. We assume $\text{Cov}_{\text{in}}(S_{1},S_{2})\geq\text{Cov}_{\text{out}}(S_{1},S_{2})$

By definition, covariance is a measure of the degree to which two variables change together. Assumption 1 means that two confidence scores will more likely produce correlated values when encountering ID data rather than OOD samples. To validate Assumption 1, we will use two classifiers that can also be employed for OOD detection: the well-known MLS [Hendrycks and Gimpel(2016)] classifier (classifier-based) and the Nearest-Mean-of-Exemplars (NME) classifier (distance-based) proposed by [Rebuffi et al.(2017)Rebuffi, Kolesnikov, Sperl, and Lampert]. To ensure consistency with Equation˜1, we will adapt the NME classifier to $\text{NME}^{+}$, defined as follows $\text{NME}^{+}(\mathbf{x})=\max(\frac{1}{\text{softmax}(\frac{\text{NME}(\mathbf{x})}{T})})$, where $T$ is a temperature parameter that controls softmax smoothness. For more details about $\text{NME}^{+}$, please refer to Appendix˜B. Since $\text{NME}^{+}$ and MLS are two scoring functions that can be easily derived from classification tasks, we introduce a piecewise function called COnsistency (CO), along with its corresponding scoring function, $\text{CO}^{+}$, as follows:

[TABLE]

[TABLE]

Intuitively, CO can be interpreted as a proxy for the covariance between $\text{NME}^{+}$ and MLS, as it measures agreement between their class predictions (denoted by $c$), while $\text{CO}^{+}$ is a scoring function derived from CO. In $\text{CO}^{+}$, the parameter $\lambda\geq 1$ controls the influence of the prediction consistency between logits and features on the final score. Fig.˜3 presents the consistency ratio for ID and OOD datasets across various models. From these results, we observe that $\text{NME}^{+}$ and MLS are more likely to make similar predictions when encountering ID data, compared to OOD data, which empirically validates Assumption 1. For simplicity, we use $\text{NME}^{+}$ and MLS as confidence scores in the definition of CO. However, as demonstrated in Section˜B.4, the same reasoning applies equally well to other confidence scores.

3.2 Motivation

Current post-hoc OOD detection methods can be divided into two main categories: feature truncation [Sun et al.(2021)Sun, Guo, and Li, Sun and Li(2022)] and scoring functions [Hendrycks and Gimpel(2016), Liu et al.(2020)Liu, Wang, Owens, and Li]. Feature truncation methods aim to increase the gap between ID and OOD samples through scaling, pruning or clipping operations on features. However, feature truncation alone cannot perform OOD detection and is generally paired with simple scoring functions such as MLS [Hendrycks and Gimpel(2016)] or Energy [Liu et al.(2020)Liu, Wang, Owens, and Li]. Scoring functions use various types of inputs, such as logits or features, to produce a score that can distinguish between ID and OOD samples. Recent advancements [Wang et al.(2022)Wang, Li, Feng, and Zhang, Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang, Park et al.(2023)Park, Jung, and Teoh] suggest using multiple input sources to better handle the diversity of OOD samples. These observations lead us to the following questions:

Does combining multiple scoring functions improve OOD detection? 2. 2.

Can feature truncation methods improve the performance of recent scoring functions?

In this paper, we answer these questions through both theoretical and empirical analysis.

3.3 Theoretical analysis

Combining multiple scoring function. Recent state-of-the-art OOD detection methods have explored the benefits of combining feature and logits information [Wang et al.(2022)Wang, Li, Feng, and Zhang, Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang] to better handle the diversity of OOD samples. To effectively merge these two sources of input, these methods typically combine the Energy score (logits-based information) with a custom scoring function based on features to achieve state-of-the-art performance. Building on these insights, we make the following proposition:

Proposition 1**.**

Let $n\in\mathbb{N}$ with $n\geq 2$, and let $S_{1},\ldots,S_{n}$ be distinct scoring functions. Under Assumption 1 and using Equation˜2, The inequality below is satisfied:

[TABLE]

Simplified proof. The full proof is in Appendix˜A. For simplicity, we consider the case where $n=2$, as the case $n>2$ can be trivially derived by recurrence. Using the mathematical expression of the covariance and the scalability properties of a scoring function, we obtain:

[TABLE]

By symmetry, we have $\mathbb{V}(S_{1}\cdot S_{2})\geq\mathbb{V}(S_{2})$, which leads to $\mathbb{V}(S_{1}\cdot S_{2})\geq\max(\mathbb{V}(S_{1}),\mathbb{V}(S_{2}))$

Proposition˜1 states that combining multiple scoring functions results in a scoring function that separates ID and OOD data at least as effectively as the best individual scoring function, thus answering our first question.

Pairing feature truncation with scoring function. The second question is partially addressed in the literature. For instance, VRA [Xu et al.(2023b)Xu, Lian, Liu, and Tao] demonstrates that feature truncation increases the gap between ID and OOD distributions, as shown in Equation˜3. Additionally, ReAct [Sun et al.(2021)Sun, Guo, and Li] shows that clipping high activation values reduces both feature and logits activations more for OOD samples than for ID samples, leading to a greater separation between ID and OOD. Feature truncation methods [Sun et al.(2021)Sun, Guo, and Li, Xu et al.(2023b)Xu, Lian, Liu, and Tao, Sun and Li(2022), Xu et al.(2023a)Xu, Chen, Franchi, and Yao] indicate that this increased separation also transfers to the logits space and energy-based scores when combined with simple scoring functions, such as the Energy score [Liu et al.(2020)Liu, Wang, Owens, and Li] or MLS [Hendrycks and Gimpel(2016)]. While recent scoring functions are more complex than Energy or MLS, they all rely on either features or logits as inputs. Based on these observations, we make the following hypothesis:

Hypothesis 1**.**

Let g be a feature truncation method such that $\mathbb{E}_{\text{in}}[g(z)]-\mathbb{E}_{\text{out}}[g(z)]\geq\mathbb{E}_{\text{in}}[z]-\mathbb{E}_{\text{out}}[z]$ and let $S$ be a scoring function. The following inequality holds:

[TABLE]

While ˜1 is central to our method, a general proof is elusive due to the nature of the scoring function $S$, which typically relies on the output of a neural network. Consequently, $S$ often does not satisfy properties such as monotonicity, Lipschitz continuity, or convexity. These conditions are commonly required to apply standard analytical tools like Jensen’s inequality or continuity bounds. As a result, we treat this statement as a working hypothesis. We support it empirically in Figure˜3, which shows that recent state-of-the-art scoring functions consistently benefit from feature truncation methods. It is important to note that Mahalanobis [Lee et al.(2018)Lee, Lee, Lee, and Shin] is the only tested scoring function that does not benefit from feature truncation. This can be explained by the fact that, as a distance-based method, Mahalanobis relies on accurate feature distance values, which may be degraded by feature truncation.

3.4 MME implementation

Based on Proposition˜1 and ˜1, we have shown that various feature truncation methods and scoring functions can be combined to create a single scoring function maximizing the gap between ID and OOD. However, although it is theoretically possible to combine an infinite number of scoring functions, this approach is impractical due to computational overhead and does not necessarily guarantee better performance. Therefore, we focus on recent state-of-the-art feature truncation methods, namely SCALE [Xu et al.(2023a)Xu, Chen, Franchi, and Yao] and VRA [Xu et al.(2023b)Xu, Lian, Liu, and Tao], along with scoring functions such as fDBD [Liu and Qin(2023)], PCA [Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang] and ViM [Wang et al.(2022)Wang, Li, Feng, and Zhang], as well as our proposed $\text{NME}^{+}$ and $\text{CO}^{+}$ functions, to construct MME scoring function. The MME function is defined as follows:

[TABLE]

To ensure positivity of MME, we apply the exponential function to ${\scriptsize\text{SCALE}(\mathbf{x})-\text{ViM}(\text{VRA}(\mathbf{x}))}$. Note that $\text{NME}^{+}$ is the only scoring function that does not use feature truncation, as it is a distance-based method, as suggested by Figure˜3. Further details about the choice of combinations will be provided in Appendix˜C. A complete overview of MME pipeline is provided by Figure˜4.

4 Experiments

4.1 Experimental settings

Datasets MME was evaluated on three OOD detection benchmarks, each with training ID, test ID, and multiple OOD sets that don’t class overlap with the ID sets. For small-scale benchmarks, CIFAR-10 and CIFAR-100 [Krizhevsky et al.(2009)Krizhevsky, Hinton, et al.] serve as ID sets, with six OOD sets: LSUN-Crop [Yu et al.(2015)Yu, Seff, Zhang, Song, Funkhouser, and Xiao], LSUN-Resize [Yu et al.(2015)Yu, Seff, Zhang, Song, Funkhouser, and Xiao], iSUN [Xu et al.(2015)Xu, Ehinger, Zhang, Finkelstein, Kulkarni, and Xiao], SVHN [Netzer et al.(2011)Netzer, Wang, Coates, Bissacco, Wu, Ng, et al.], Places365 [Zhou et al.(2017)Zhou, Lapedriza, Khosla, Oliva, and Torralba], and Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi]. For large-scale OOD detection, ImageNet-1K [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] is the ID set, with iNaturalist [Van Horn et al.(2018)Van Horn, Mac Aodha, Song, Cui, Sun, Shepard, Adam, Perona, and Belongie], Places [Zhou et al.(2017)Zhou, Lapedriza, Khosla, Oliva, and Torralba], SUN [Xiao et al.(2010)Xiao, Hays, Ehinger, Oliva, and Torralba], and Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi] as OOD sets. Following recent studies [Xu et al.(2023a)Xu, Chen, Franchi, and Yao, Zhang et al.(2023)Zhang, Yang, Wang, Wang, Lin, Zhang, Sun, Du, Li, Liu, et al.], ImageNet-1K [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] is used for near-OOD (NINCO [Bitterwolf et al.(2023)Bitterwolf, Mueller, and Hein], SSB-hard [Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman]) and far-OOD (iNaturalist [Van Horn et al.(2018)Van Horn, Mac Aodha, Song, Cui, Sun, Shepard, Adam, Perona, and Belongie], Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi], OpenImage-O [Wang et al.(2022)Wang, Li, Feng, and Zhang]) detection. Refer to Appendix˜D for dataset details.

Models Following previous studies [Wang et al.(2022)Wang, Li, Feng, and Zhang, Liu et al.(2023)Liu, Lochman, and Zach, Liang et al.(2017)Liang, Li, and Srikant], we used ViT-B/16 [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] (Transformer-based) and BiT-S-R101x1 [Kolesnikov et al.(2020)Kolesnikov, Beyer, Zhai, Puigcerver, Yung, Gelly, and Houlsby] (CNN-based) as the classifier backbones for the ImageNet-1K benchmarks, while DenseNet-BC-100 [Huang et al.(2017)Huang, Liu, Van Der Maaten, and Weinberger] was used for the CIFAR benchmarks.

Methods To ensure a fair comparison, we compare MME with several state-of-the-art post-hoc OOD detectors. We consider 5 feature truncation methods combined with the Energy score: ReAct [Sun et al.(2021)Sun, Guo, and Li], DICE [Sun and Li(2022)], ASH-S [Djurisic et al.(2022)Djurisic, Bozanic, Ashok, and Liu], SCALE [Xu et al.(2023a)Xu, Chen, Franchi, and Yao] and VRA [Xu et al.(2023b)Xu, Lian, Liu, and Tao]. We also evaluate 11 scoring functions: MSP [Hendrycks and Gimpel(2016)], MLS [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song, Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman], Energy [Liu et al.(2020)Liu, Wang, Owens, and Li], Mahalanobis [Lee et al.(2018)Lee, Lee, Lee, and Shin] (Maha), KL Matching [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song] (KL), ViM [Wang et al.(2022)Wang, Li, Feng, and Zhang], SHE [Zhang et al.(2022)Zhang, Fu, Chen, Du, Li, Wang, Han, Zhang, et al.], GEN [Liu et al.(2023)Liu, Lochman, and Zach], PCA [Guan et al.(2023)Guan, Liu, Zheng, Zhou, and Wang], NNGuide [Park et al.(2023)Park, Jung, and Teoh], and fDBD [Liu and Qin(2023)]. We use the same hyperparameters as in the original papers and reproduce results using their code. Experiments were conducted on an NVIDIA GeForce RTX 3090, with CUDA 11.0 and PyTorch 1.7. In the following experiments, $T=0.5$ for the ImageNet-1k benchmark and $T=0.1$ for the CIFAR benchmark, while $\lambda$ is set to 2 for both benchmarks. MME is robust to $T$ and $\lambda$, requiring minimal hyperparameter tuning, as shown in Section˜C.3.

Evaluation metrics We report two metrics: AUROC, the area under the ROC curve (higher is better), and FPR95 (false positive rate at 95% true positive rate, lower is better). Both metrics are presented as percentages, with the best results in bold.

4.2 Evaluation on ImageNet-1k

On the ImageNet-1K large-scale benchmark, Table˜2 summarizes the OOD detection performance of each method across different OOD datasets, as well as the average performance over all four datasets. With the BiT backbone, MME outperforms all methods on three out of the four datasets in terms of FPR95, achieving an average FPR95 of 27.57%, an improvement of 6% over the best existing baseline. Furthermore, with the ViT model, MME achieves the highest performance on the Textures dataset and the best average performance overall. On the other three datasets, MME consistently delivers competitive results, generally ranking among the top methods (second or third), further highlighting its robustness.

Across all experiments, at least nine different methods achieve a top-three performance on a specific model and dataset, highlighting the sensitivity of existing OOD detectors to both the model architecture and dataset characteristics. In contrast, MME consistently ranks among the top three, demonstrating that our ensemble strategy enhances robustness and improves the detection of various types of OOD samples.

4.3 Evaluation on CIFAR benchmarks

On small-scale CIFAR benchmarks, the ensembling strategy also contributes to achieving state-of-the-art performance with greater consistency. Table˜1 presents the average performance across six OOD datasets for both CIFAR-10 and CIFAR-100 as ID sets, demonstrating that MME outperforms existing post-hoc detectors. Notably, while ViM achieves the second-best performance on both CIFAR-10 and CIFAR-100 benchmarks and even comes within 1% of MME in terms of FPR95 on CIFAR-10, it falls significantly short on CIFAR-100. In this case, MME achieves an FPR95 of 28.96%, offering a substantial improvement of nearly 10% over ViM, which records an FPR95 of 38.36%.

4.4 Evaluation on near- and far-OOD

Figure˜1 illustrates the trade-off between near-OOD and far-OOD detection performance across various state-of-the-art methods. As highlighted by [Park et al.(2023)Park, Jung, and Teoh], simple probability-based methods such as MLS [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song] and KL Matching [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song] achieve competitive results on near-OOD but perform poorly on far-OOD. In contrast, distance-based scoring functions such as Mahalanobis [Lee et al.(2018)Lee, Lee, Lee, and Shin] struggle with near-OOD detection due to the difficulty in defining clear decision boundaries. Recent methods like fDBD [Liu and Qin(2023)] and ViM [Wang et al.(2022)Wang, Li, Feng, and Zhang], which integrate multiple input sources, offer a better balance between near-OOD and far-OOD performance. Additionally, feature truncation techniques such as SCALE [Xu et al.(2023a)Xu, Chen, Franchi, and Yao] and ASH-S [Djurisic et al.(2022)Djurisic, Bozanic, Ashok, and Liu] achieve a strong trade-off due to their effective scaling properties. Ultimately, MME outperforms all other approaches on both near-OOD and far-OOD tasks, further emphasizing the advantage of combining feature truncation with multiple scoring functions to detect a wide variety of OOD samples.

5 Conclusion

In this study, we analyzed existing feature truncation and scoring function methods and demonstrated both theoretically and empirically that combining these methods improves OOD detection performance. Based on this insight, we proposed a MME strategy that fuses state-of-the-art OOD detectors to better handle the diversity of OOD samples. MME has been thoroughly tested on both large and small-scale benchmarks, considering both near- and far-OOD scenarios. MME has been shown to achieve state-of-the-art performance across various models and datasets, demonstrating its effectiveness and robustness in detecting a wide variety of OOD samples.

Appendix A Mathematical proof

Here, we provide the proof of Proposition˜1.

Proposition 1 Let $n\in\mathbb{N}$ such that $n\geq 2$. Let $S_{1},\ldots,S_{n}$ different scoring functions. $\mathbb{V}(S_{1}\cdot\ldots\cdot S_{n})\geq\max(\mathbb{V}(S_{1}),\ldots,\mathbb{V}(S_{n}))$

Proof. We start with the case where $n=2$. By definition, we have :

[TABLE]

However, thanks to Assumption 1, we have $\text{Cov}_{\text{in}}(S_{1},S_{2})\geq\text{Cov}_{\text{out}}(S_{1},S_{2})$ resulting in,

[TABLE]

By definition of a scoring function and Eq.˜2, we have $\mathbb{E}_{\text{in}}[S_{2}(\mathbf{x})]\geq\mathbb{E}_{\text{out}}[S_{2}(\mathbf{x})]$ resulting in,

[TABLE]

By definition, a scoring function is invariant to scaling, therefore if $\mathbb{E}_{\text{out}}(S_{2}(\mathbf{x}))<1$, we could easily find a $\lambda$ such as $\mathbb{E}_{\text{out}}(\lambda S_{2}(\mathbf{x}))\geq 1$. For convenience and clarity, we will suppose $\mathbb{E}_{\text{out}}(S_{2}(\mathbf{x}))\geq 1$ ultimately leading us to

[TABLE]

The same methodology can be applied to obtain $\mathbb{V}(S_{1}\cdot S_{2})\geq\mathbb{V}(S_{2})$, ultimately leading to $\mathbb{V}(S_{1}\cdot S_{2})\geq\max(\mathbb{V}(S_{1}),\mathbb{V}(S_{2}))$.

For the case $n>2$, we can trivially obtain the result by recurrence by defining $S_{1}=S_{1}\cdot\ldots\cdot S_{n-1}$ and $S_{2}=S_{n}$ which completes the proof.

Appendix B Detailed Description of $\text{NME}^{+}$ and CO

B.1 Motivation of $\text{NME}^{+}$ and CO

The first OOD detectors, MSP [Hendrycks and Gimpel(2016)] and MLS [Hendrycks et al.(2019)Hendrycks, Basart, Mazeika, Zou, Kwon, Mostajabi, Steinhardt, and Song, Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman], rely directly on the output of the classifier. The underlying assumption is that in-distribution samples are more likely to receive higher confidence scores than OOD samples, as the model is generally more "confident" in its predictions for familiar data. This principle allows us to easily construct an OOD detector based on any classifier following the same logic.

While MSP and MLS are classifier-based methods, we propose using the Nearest-Mean-of-Exemplars (NME) classifier, a distance-based approach originally introduced in [Rebuffi et al.(2017)Rebuffi, Kolesnikov, Sperl, and Lampert] to better leverage feature representations. Furthermore, as discussed in Section˜2, classifier-based and distance-based OOD detectors are complementary. Therefore, combining both approaches through the CO function can enhance OOD detection performance across a broader range of scenarios and allows us to empirically validate Assumption 1.

B.2 Implementation of $\text{NME}^{+}$

Let $\phi$ be the feature extractor. For each class $y\in\mathcal{Y}$, the mean feature vector is computed as:

[TABLE]

Given a test input $x\in\mathcal{X}$, its feature representation $\phi(x)$ is compared to all class means, and the predicted label is assigned as:

[TABLE]

To ensure positivity and remain consistent with the definition of scoring function and Equation˜1, we use $\text{NME}^{+}$ as a scoring function.

[TABLE]

B.3 Performance of $\text{NME}^{+}$ and $\text{CO}^{+}$

Table˜3 presents the raw performance of $\text{NME}^{+}$ and $\text{CO}^{+}$ compared to the full MME framework and other state-of-the-art methods. Despite its relative simplicity, $\text{NME}^{+}$ achieves competitive results, comparable to Mahalanobis and even GEN. When combined with $\text{CO}^{+}$, it yields a 1.26% improvement in average FPR95, highlighting the benefit of integrating MLS with NME-based predictions. Furthermore, as demonstrated in Section˜4.2, incorporating both $\text{NME}^{+}$ and $\text{CO}^{+}$ into other state-of-the-art approaches to form MME leads to superior performance and enhanced robustness across diverse OOD scenarios.

B.4 CO performance with various scoring function

In Section˜3.1 and Fig.˜3, we validate Assumption 1 using the CO function (Equation˜4). The CO function is designed for OOD detectors that also provide classification predictions, enabling direct comparison between predictions. Extending CO to modern scoring functions is non-trivial, as these typically do not produce classification outputs. Therefore, to further validate Assumption 1, we measure the covariance between six scoring functions on the ID dataset, as well as the average covariance across four OOD datasets. As shown in Table˜4, the covariance values for the ID dataset are consistently higher than those for the OOD datasets, which further supports Assumption 1 and Proposition˜1.

Appendix C Ablation study

C.1 Varying feature truncation

Table˜5 illustrates how MME performance changes when different combinations of feature truncation and scoring functions are applied. Overall, we observe that the highest average performance is achieved with the default parameters. However, performance remains consistently stable, demonstrating the robustness of our ensemble approach.

C.2 Adding more scoring function

Table˜6 shows the impact of adding more scoring functions to MME. Empirically, we observe that adding more functions no longer improves performance. As explained by Proposition˜1, adding more functions does not necessarily enhance performance. In fact, adding more functions without novel information could compromise Assumption 1, leading to a degradation in performance.

C.3 Effect of hyperparameter

We empirically analyze how the performance of MME varies with different values of $\lambda$ and $T$ in terms of AUROC.

First, we examine the effect of $\lambda$. The first row of Figure˜5 shows that the best performance is achieved with $\lambda=1.5$ for both ViT [Dosovitskiy et al.(2020)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, et al.] and BiT [Kolesnikov et al.(2020)Kolesnikov, Beyer, Zhai, Puigcerver, Yung, Gelly, and Houlsby] architectures. However, once this value increases further, the performance remains unchanged, indicating that MME is relatively robust to the choice of $\lambda$.

We also investigate the effect of different values of $T$. The second row of Figure˜5 demonstrates that very low values of $T$ ($T<0.1$) lead to suboptimal performance. Performance improves as $T$ increases, reaching its peak at $T=0.5$. Beyond this point, further increases in $T$ result in a performance plateau or a slight decline, suggesting that excessive smoothing of predictions is unnecessary.

The current evaluation protocol for OOD detection relies on the test dataset, which may not be ideal for real-world applications. However, as shown in Figure˜5, MME remains highly robust to variations in hyperparameters due to its ensemble strategy. Unless $\lambda$ or $T$ undergoes extreme changes, performance remains stable, with less than a 2% difference between the best and worst cases. Additionally, MME was evaluated on completely unseen datasets, including NINCO [Bitterwolf et al.(2023)Bitterwolf, Mueller, and Hein] and SSB-hard [Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman] for near-OOD detection and OpenImage-O [Wang et al.(2022)Wang, Li, Feng, and Zhang] for far-OOD detection and it consistently achieves state-of-the-art performance, as illustrated in Figure˜1.

Appendix D Dataset description

Large-scale. For large-scale evaluation, we use ImageNet-1k [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] as the in-distribution dataset and evaluate on four OOD datasets following the setup proposed by

• •

iNaturalist [Van Horn et al.(2018)Van Horn, Mac Aodha, Song, Cui, Sun, Shepard, Adam, Perona, and Belongie] dataset contains 859,000 images of plants and animals, covering over 5,000 different species. Each image is resized to have a maximum dimension of 800 pixels. For evaluation, we use a random sample of 10,000 images from 110 classes that do not overlap with those in ImageNet-1k.

• •

Places [Zhou et al.(2017)Zhou, Lapedriza, Khosla, Oliva, and Torralba] dataset is a scene dataset. For evaluation, we use a selected subset of 10,000 images spanning 50 classes that are not included in ImageNet-1k.

• •

SUN [Xiao et al.(2010)Xiao, Hays, Ehinger, Oliva, and Torralba] dataset comprises over 130,000 images of scenes across 397 categories. Although there are overlapping categories between SUN and ImageNet-1k, our evaluation uses 10,000 images randomly sampled from 50 classes that do not overlap with ImageNet-1k labels.

• •

Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi] dataset includes 5,640 real-world texture images categorized into 47 different classes. We utilize the entire dataset for our evaluation.

Small-scale. The CIFAR [Krizhevsky et al.(2009)Krizhevsky, Hinton, et al.] benchmarks, CIFAR-10 and CIFAR-100, are widely recognized ID datasets in the literature, containing 10 and 100 classes, respectively. We follow the standard split, using 50,000 training images and 10,000 test images. Our approach is evaluated using six common OOD datasets, which are listed below:

• •

SVHN [Netzer et al.(2011)Netzer, Wang, Coates, Bissacco, Wu, Ng, et al.] consists of color images of house numbers, categorized into ten digit classes (0-9). We use the entire test set, which includes 26,032 images.

• •

LSUN-Crop [Yu et al.(2015)Yu, Seff, Zhang, Song, Funkhouser, and Xiao] comprises 10,000 testing images across 10 different scenes. For evaluation, image patches of size 32$\times$32 are randomly cropped from this dataset.

• •

LSUN-Resize [Yu et al.(2015)Yu, Seff, Zhang, Song, Funkhouser, and Xiao] comprises 10,000 testing images across 10 different scenes. For evaluation, image patches of size 32$\times$32 are randomly resized from this dataset.

• •

iSUN [Xu et al.(2015)Xu, Ehinger, Zhang, Finkelstein, Kulkarni, and Xiao] provides ground truth gaze traces on images from the SUN dataset. Following prior work, we use a subset of 8,925 images for our evaluation.

• •

Places365 [Zhou et al.(2017)Zhou, Lapedriza, Khosla, Oliva, and Torralba] is a large-scale dataset including photographs of scenes categorized into 365 scene types, with 900 images per category in the test set, totaling 328,500 images. We utilize the entire test set for our evaluation.

• •

Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi] dataset includes 5,640 real-world texture images categorized into 47 different classes. We utilize the entire dataset for our evaluation.

Near and far-OOD. Recent works [Xu et al.(2023a)Xu, Chen, Franchi, and Yao, Zhang et al.(2023)Zhang, Yang, Wang, Wang, Lin, Zhang, Sun, Du, Li, Liu, et al.] differentiate between near-OOD and far-OOD cases. To remain consistent with these studies, we adopt their settings: ImageNet-1K is used as the ID set while the following datasets are used for near-OOD evaluation:

• •

SSB-hard [Vaze et al.(2021)Vaze, Han, Vedaldi, and Zisserman] is the hard split of the Semantic Shift Benchmark (SSB), which is more aligned with detecting semantic novelty as opposed to other distributional shifts addressed by related fields such as out-of-distribution detection. We use the entire test set comprising 49,000 images.

• •

NINCO [Bitterwolf et al.(2023)Bitterwolf, Mueller, and Hein] is a dataset containing images that do not include objects from any of the 1000 classes of ImageNet-1K, making it suitable for evaluating out-of-distribution detection on ImageNet-1K. The NINCO dataset consists of 64 OOD classes with a total of 5,879 samples. Each sample was individually inspected by the authors to ensure it does not contain ID objects.

The following datasets are used for far-OOD evaluation:

• •

iNaturalist [Van Horn et al.(2018)Van Horn, Mac Aodha, Song, Cui, Sun, Shepard, Adam, Perona, and Belongie] dataset contains 859,000 images of plants and animals, covering over 5,000 different species. Each image is resized to have a maximum dimension of 800 pixels. For evaluation, we use a random sample of 10,000 images from 110 classes that do not overlap with those in ImageNet-1k.

• •

Textures [Cimpoi et al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi] dataset includes 5,640 real-world texture images categorized into 47 different classes. We utilize the entire dataset for our evaluation.

• •

OpenImage-O [Wang et al.(2022)Wang, Li, Feng, and Zhang] dataset is curated image-by-image from the test set of OpenImage-V3, collected from Flickr. It does not have a predefined list of class names or tags, which results in natural class statistics and avoids initial design bias.