CFOF: A Concentration Free Measure for Anomaly Detection

TL;DR

The paper introduces CFOF, a new outlier measure that remains effective in high dimensions, is theoretically immune to concentration effects, and is computationally efficient for large datasets.

Contribution

It formalizes CFOF as a concentration-free outlier measure, provides its distribution in high dimensions, and introduces a scalable algorithm for outlier detection.

Findings

CFOF does not concentrate in high dimensions.

CFOF scores depend on squared norm and kurtosis.

The fast-CFOF algorithm is efficient and scalable.

Abstract

We present a novel notion of outlier, called the Concentration Free Outlier Factor, or CFOF. As a main contribution, we formalize the notion of concentration of outlier scores and theoretically prove that CFOF does not concentrate in the Euclidean space for any arbitrary large dimensionality. To the best of our knowledge, there are no other proposals of data analysis measures related to the Euclidean distance for which it has been provided theoretical evidence that they are immune to the concentration effect. We determine the closed form of the distribution of CFOF scores in arbitrarily large dimensionalities and show that the CFOF score of a point depends on its squared norm standard score and on the kurtosis of the data distribution, thus providing a clear and statistically founded characterization of this notion. Moreover, we leverage this closed form to provide evidence that the definition does not suffer of the hubness problem affecting other measures. We prove that the number of CFOF outliers coming from each cluster is proportional to cluster size and kurtosis, a property that we call semi-locality. We determine that semi-locality characterizes existing reverse nearest neighbor-based outlier definitions, thus clarifying the exact nature of their observed local behavior. We also formally prove that classical distance-based and density-based outliers concentrate both for bounded and unbounded sample sizes and for fixed and variable values of the neighborhood parameter. We introduce the fast-CFOF algorithm for detecting outliers in large high-dimensional dataset. The algorithm has linear cost, supports multi-resolution analysis, and is embarrassingly parallel. Experiments highlight that the technique is able to efficiently process huge datasets and to deal even with large values of the neighborhood parameter, to avoid concentration, and to obtain excellent accuracy.

Figures

Tables

Table 1. Table 1. MIMD speedup of parallel fast - CFOF for different sample sizes s 𝑠 s ( ∗ the last speedup of MNIST is relative to whole dataset, i.e. to the sample size s = 60 , 000 𝑠 60 000 s=60,\!000 , and is not considered in the average speedup).

Dataset / $s$	$512$	$3,584$	$15,360$	$26,624$	$120,320$
MNIST	$4.6407$	$6.4068$	$7.3548$	$7.5545$	${}_{}{}^{\ast }7.6404$
MSD	$4.8205$	$6.3137$	$7.3426$	$7.5671$	$7.8296$
SIFT10M	$4.8265$	$6.3677$	$7.3545$	$7.5998$	$7.8325$
Clust2	$4.8381$	$6.7250$	$7.4681$	$7.6165$	$7.8330$
Average	$4.7815$	$6.4533$	$7.3800$	$7.5845$	$7.8317$

Table 2. Table 2. MNIST dataset: P ​ r ​ e ​ c ​ @ ​ n 𝑃 𝑟 𝑒 𝑐 @ 𝑛 Prec@n .

$Prec@0.001$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.2333$ $0.4500$ $0.7167$ $0.7167$ $3584$ $0.6333$ $0.6667$ $0.7833$ $0.9333$ $0.9000$ $15360$ $0.8333$ $0.9167$ $0.9333$ $0.9833$ $0.9833$ $26624$ $0.9000$ $0.9500$ $0.9667$ $0.9667$ $0.9833$	$Prec@0.005$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.4400$ $0.5700$ $0.7133$ $0.7567$ $3584$ $0.5733$ $0.7533$ $0.8033$ $0.9367$ $0.9400$ $15360$ $0.7733$ $0.9233$ $0.9367$ $0.9733$ $0.9867$ $26624$ $0.8900$ $0.9467$ $0.9633$ $0.9900$ $0.9867$
$Prec@0.01$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.4433$ $0.5667$ $0.7467$ $0.7583$ $3584$ $0.5517$ $0.7500$ $0.8200$ $0.9250$ $0.9167$ $15360$ $0.8117$ $0.9050$ $0.9300$ $0.9633$ $0.9650$ $26624$ $0.8633$ $0.9350$ $0.9600$ $0.9633$ $0.9717$	$Prec@0.05$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.5353$ $0.6427$ $0.7827$ $0.8430$ $3584$ $0.6490$ $0.8057$ $0.8470$ $0.9277$ $0.9490$ $15360$ $0.8443$ $0.9260$ $0.9497$ $0.9683$ $0.9787$ $26624$ $0.8890$ $0.9553$ $0.9643$ $0.9777$ $0.9817$

Table 3. Table 3. MSD dataset: P ​ r ​ e ​ c ​ @ ​ n 𝑃 𝑟 𝑒 𝑐 @ 𝑛 Prec@n .

$Prec@0.001$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.7771$ $0.8411$ $0.8643$ $0.8411$ $3584$ $0.7829$ $0.9574$ $0.9690$ $1.0000$ $1.0000$ $15360$ $0.9128$ $0.9729$ $0.9787$ $1.0000$ $1.0000$ $26624$ $0.9341$ $0.9787$ $0.9826$ $1.0000$ $1.0000$	$Prec@0.005$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0062$ $0.6923$ $0.8196$ $0.8514$ $0.8157$ $3584$ $0.7439$ $0.8956$ $0.9274$ $0.9988$ $0.9992$ $15360$ $0.8867$ $0.9519$ $0.9697$ $0.9988$ $0.9992$ $26624$ $0.9208$ $0.9674$ $0.9794$ $0.9984$ $0.9988$
$Prec@0.01$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.6962$ $0.7965$ $0.9523$ $0.9816$ $3584$ $0.7396$ $0.8948$ $0.9265$ $0.9957$ $1.0000$ $15360$ $0.8840$ $0.9542$ $0.9643$ $0.9986$ $1.0000$ $26624$ $0.9191$ $0.9664$ $0.9785$ $0.9983$ $1.0000$	$Prec@0.05$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0512$ $0.6891$ $0.7988$ $0.9089$ $0.9401$ $3584$ $0.7418$ $0.8920$ $0.9272$ $0.9680$ $0.9809$ $15360$ $0.8852$ $0.9527$ $0.9700$ $0.9837$ $0.9907$ $26624$ $0.9173$ $0.9675$ $0.9806$ $0.9876$ $0.9934$

Table 4. Table 4. SIFT10M dataset: P ​ r ​ e ​ c ​ @ ​ n 𝑃 𝑟 𝑒 𝑐 @ 𝑛 Prec@n .

$Prec@0.001$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.2959$ $0.4381$ $0.7260$ $0.8462$ $3584$ $0.4495$ $0.6248$ $0.7247$ $0.8970$ $0.9653$ $15360$ $0.7126$ $0.8191$ $0.8738$ $0.9577$ $0.9927$ $26624$ $0.7897$ $0.8699$ $0.9135$ $0.9684$ $0.9964$	$Prec@0.005$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0053$ $0.3861$ $0.5693$ $0.8418$ $0.8941$ $3584$ $0.4920$ $0.7085$ $0.8099$ $0.9483$ $0.9705$ $15360$ $0.7448$ $0.8622$ $0.9166$ $0.9840$ $0.9912$ $26624$ $0.8124$ $0.9016$ $0.9417$ $0.9903$ $0.9942$
$Prec@0.01$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.4221$ $0.6116$ $0.8603$ $0.9086$ $3584$ $0.5096$ $0.7431$ $0.8359$ $0.9490$ $0.9703$ $15360$ $0.7558$ $0.8859$ $0.9257$ $0.9781$ $0.9888$ $26624$ $0.8209$ $0.9206$ $0.9468$ $0.9850$ $0.9930$	$Prec@0.05$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0510$ $0.4887$ $0.6527$ $0.8754$ $0.9173$ $3584$ $0.5784$ $0.7926$ $0.8659$ $0.9585$ $0.9730$ $15360$ $0.7989$ $0.9099$ $0.9437$ $0.9840$ $0.9899$ $26624$ $0.8542$ $0.9377$ $0.9611$ $0.9896$ $0.9935$

Table 5. Table 5. Clust2 dataset: P ​ r ​ e ​ c ​ @ ​ n 𝑃 𝑟 𝑒 𝑐 @ 𝑛 Prec@n .

$Prec@0.001$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.5000$ $0.5500$ $0.7600$ $0.6800$ $3584$ $0.6200$ $0.7900$ $0.8600$ $0.9300$ $0.9200$ $15360$ $0.8100$ $0.8900$ $0.9100$ $0.9600$ $0.9700$ $26624$ $0.8700$ $0.9200$ $0.9500$ $0.9800$ $1.0000$	$Prec@0.005$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0060$ $0.5840$ $0.6820$ $0.7960$ $0.7760$ $3584$ $0.6900$ $0.8420$ $0.8900$ $0.9240$ $0.9200$ $15360$ $0.8580$ $0.9500$ $0.9620$ $0.9740$ $0.9580$ $26624$ $0.8940$ $0.9560$ $0.9700$ $0.9760$ $0.9700$
$Prec@0.01$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.6010$ $0.7120$ $0.8330$ $0.8090$ $3584$ $0.6960$ $0.8520$ $0.9000$ $0.9500$ $0.9340$ $15360$ $0.8720$ $0.9350$ $0.9600$ $0.9810$ $0.9540$ $26624$ $0.8980$ $0.9620$ $0.9700$ $0.9920$ $0.9790$	$Prec@0.05$ $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ $0.0526$ $0.6788$ $0.7612$ $0.8680$ $0.8794$ $3584$ $0.7676$ $0.8864$ $0.9110$ $0.9622$ $0.9568$ $15360$ $0.9050$ $0.9544$ $0.9642$ $0.9868$ $0.9820$ $26624$ $0.9318$ $0.9690$ $0.9762$ $0.9960$ $0.9858$

Table 6. Table 6. Spearman rank correlation.

MNIST $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.6265$ $0.8022$ $0.9526$ $0.9793$ $3584$ $0.7429$ $0.9375$ $0.9669$ $0.9934$ $0.9971$ $15360$ $0.9447$ $0.9880$ $0.9938$ $0.9987$ $0.9994$ $26624$ $0.9737$ $0.9946$ $0.9972$ $0.9995$ $0.9997$	MSD $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.7757$ $0.8976$ $0.9741$ $0.9852$ $3584$ $0.8542$ $0.9712$ $0.9853$ $0.9965$ $0.9979$ $15360$ $0.9714$ $0.9941$ $0.9969$ $0.9992$ $0.9995$ $26624$ $0.9847$ $0.9968$ $0.9984$ $0.9996$ $0.9998$
SIF10M $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.6315$ $0.8022$ $0.9482$ $0.9725$ $3584$ $0.7495$ $0.9369$ $0.9665$ $0.9925$ $0.9960$ $15360$ $0.9424$ $0.9863$ $0.9927$ $0.9984$ $0.9991$ $26624$ $0.9683$ $0.9927$ $0.9961$ $0.9991$ $0.9995$	Clust2 $s$ ${\varrho }_1$ ${\varrho }_2$ ${\varrho }_3$ ${\varrho }_4$ ${\varrho }_5$ $0.001$ $0.005$ $0.01$ $0.05$ $0.1$ $512$ — $0.8725$ $0.9425$ $0.9822$ $0.9881$ $3584$ $0.9333$ $0.9860$ $0.9922$ $0.9975$ $0.9983$ $15360$ $0.9884$ $0.9972$ $0.9984$ $0.9995$ $0.9996$ $26624$ $0.9943$ $0.9986$ $0.9992$ $0.9997$ $0.9998$

Table 7. Table 7. Concentration ratio (see Equation ( 17 )) obtained by CFOF on real data projected on its d i subscript 𝑑 𝑖 d_{i} first principal components for α = 0.1 𝛼 0.1 \alpha=0.1 and ϱ = 0.01 italic-ϱ 0.01 \varrho=0.01 .

original data, $\alpha =0.1$, $\varrho =0.01$
	$d_1$	$d_2$	$d_3$	$d_4$	$d_{max}$
SIFT	$1.334$	$1.661$	$1.891$	$2.100$	$2.213$
Isolet	$1.985$	$1.933$	$3.018$	$3.496$	$3.263$
MNIST	$0.762$	$0.974$	$1.300$	$1.568$	$1.747$
Sports	$17.014$	$9.161$	$4.246$	$2.847$	$2.732$
Arcene	$1.712$	$2.387$	$3.032$	$3.100$	$2.900$
RNA-Seq	$2.063$	$3.549$	$7.354$	$5.781$	$4.834$
normalized data, $\alpha =0.1$, $\varrho =0.01$
	$d_1$	$d_2$	$d_3$	$d_4$	$d_{max}$
SIFT	$1.880$	$2.146$	$2.445$	$2.755$	$2.810$
Isolet	$23.924$	$23.914$	$23.845$	$23.830$	$23.818$
MNIST	$23.795$	$36.031$	$34.974$	$26.449$	$22.963$
Sports	$12.145$	$14.678$	$8.780$	$4.543$	$4.062$
Arcene	$21.883$	$24.090$	$22.639$	$18.310$	$19.433$
RNA-Seq	$12.858$	$15.504$	$18.314$	$12.606$	$12.297$

Table 8. Table 8. Concentration ratio (see Equation ( 17 )) obtained by different outlier detection definitions on real data in the full feature space for α = 0.1 𝛼 0.1 \alpha=0.1 and ϱ = 0.01 italic-ϱ 0.01 \varrho=0.01 .

original data, $\alpha =0.1$, $\varrho =0.01$
	${\kappa }_{orig}$	CFOF	ODIN	aHub2	aKNN	LOF	iForest
SIFT	$3.0$	$2.213$	$0.071$	$0.264$	$0.052$	$0.038$	$0.015$
Isolet	$3.2$	$3.263$		$0.050$	$0.055$	$0.055$	$0.011$
MNIST	$2.2$	$1.747$	$0.085$	$0.168$	$0.056$	$0.141$	$0.009$
Sports	$10.4$	$2.732$	$0.013$	$0.368$	$0.411$	$1.492$	$0.154$
Arcene	$2.2$	$2.900$	$0.098$	$0.163$	$0.169$	$0.064$	$0.008$
RNA-Seq	$3.3$	$4.834$		$0.548$	$0.107$	$0.093$	$0.011$
normalized data, $\alpha$ = $0.1$, $\varrho$ = $0.01$
	${\kappa }_{norm}$	CFOF	ODIN	aHub2	aKNN	LOF	iForest
SIFT	$5.7$	$2.810$	$0.066$	$0.285$	$0.054$	$0.036$	$0.012$
Isolet	$10.0$	$23.817$		$0.164$	$0.270$	$0.073$	$0.016$
MNIST	$895.8$	$22.963$	$0.077$	$0.389$	$1.178$	$1.030$	$0.009$
Sports	$14.5$	$4.0617$	$0.005$	$0.186$	$0.318$	$0.807$	$0.013$
Arcene	$24.8$	$19.433$		$0.457$	$0.284$	$0.143$	$0.009$
RNA-Seq	$14.7$	$12.297$		$0.514$	$0.141$	$0.153$	$0.011$

Table 9. Table 9. AUC and Precision for the synthetic datasets.

Unimodal dataset
$AU{C}_{mean}$ $AU{C}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.9886$ $0.9945$ $0.9957$ $0.9962$ $0.9999$ $0.9999$ $0.9998$ $0.9999$ ODIN $0.9523$ $0.8777$ $0.8331$ $0.8194$ $0.9995$ $0.9997$ $0.9997$ $0.9996$ AntiHub2 $0.9361$ $0.7802$ $0.6560$ $0.6261$ $0.9868$ $0.9639$ $0.9154$ $0.8646$ aKNN $0.9939$ $0.9934$ $0.9935$ $0.9925$ $0.9998$ $0.9999$ $0.9998$ $0.9998$ LOF $0.9725$ $0.9850$ $0.9877$ $0.9879$ $0.9999$ $0.9999$ $0.9998$ $0.9998$ FastABOD $0.9833$ $0.9317$ $0.8466$ $0.7161$ $0.9988$ $0.9990$ $0.9937$ $0.9485$ iForest — — — — $0.9790$ $0.9045$ $0.6925$ $0.5781$
$Pre{c}_{mean}$ $Pre{c}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.8321$ $0.8758$ $0.8840$ $0.8780$ $0.9760$ $0.9790$ $0.9673$ $0.9730$ ODIN $0.6292$ $0.5387$ $0.4891$ $0.4750$ $0.9380$ $0.9560$ $0.9533$ $0.9500$ AntiHub2 $0.5080$ $0.2602$ $0.1271$ $0.1038$ $0.7120$ $0.5780$ $0.4000$ $0.3180$ aKNN $0.8582$ $0.8606$ $0.8572$ $0.8424$ $0.9700$ $0.9740$ $0.9680$ $0.9660$ LOF $0.7614$ $0.8171$ $0.8283$ $0.8195$ $0.9700$ $0.9720$ $0.9620$ $0.9680$ FastABOD $0.7750$ $0.6119$ $0.3941$ $0.1907$ $0.9080$ $0.9200$ $0.7980$ $0.5140$ iForest — — — — $0.7160$ $0.4140$ $0.1480$ $0.0800$
Multimodal dataset
$AU{C}_{mean}$ $AU{C}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.9730$ $0.9851$ $0.9837$ $0.9825$ $0.9988$ $0.9989$ $0.9989$ $0.9989$ ODIN $0.9317$ $0.8346$ $0.8089$ $0.7972$ $0.9987$ $0.9988$ $0.9987$ $0.9989$ AntiHub2 $0.8981$ $0.7218$ $0.6192$ $0.5802$ $0.9727$ $0.9649$ $0.8931$ $0.8284$ aKNN $0.7582$ $0.7584$ $0.7584$ $0.7582$ $0.7614$ $0.7622$ $0.7621$ $0.7621$ LOF $0.9512$ $0.9605$ $0.9642$ $0.9632$ $0.9961$ $0.9988$ $0.9987$ $0.9989$ FastABOD $0.7513$ $0.7286$ $0.6785$ $0.6204$ $0.7588$ $0.7619$ $0.7620$ $0.7621$ iForest — — — — $0.7457$ $0.6889$ $0.6005$ $0.5461$
$Pre{c}_{mean}$ $Pre{c}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.7566$ $0.7979$ $0.8125$ $0.8047$ $0.9097$ $0.9130$ $0.9229$ $0.9183$ ODIN $0.5891$ $0.4919$ $0.4725$ $0.4576$ $0.8930$ $0.9113$ $0.9130$ $0.9128$ AntiHub2 $0.4358$ $0.2154$ $0.1334$ $0.1066$ $0.6220$ $0.6580$ $0.5000$ $0.4040$ aKNN $0.4814$ $0.4819$ $0.4843$ $0.4858$ $0.4980$ $0.5000$ $0.5000$ $0.5000$ LOF $0.6715$ $0.7320$ $0.7530$ $0.7473$ $0.8540$ $0.9060$ $0.9140$ $0.9160$ FastABOD $0.4490$ $0.3847$ $0.2722$ $0.1656$ $0.4860$ $0.5000$ $0.5000$ $0.5000$ iForest — — — — $0.4100$ $0.2920$ $0.1140$ $0.0620$
Multimodal artificial dataset
$AU{C}_{mean}$ $AU{C}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.9834$ $0.9999$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ ODIN $0.9408$ $0.8631$ $0.8147$ $0.8048$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ antiHub2 $0.9365$ $0.8425$ $0.7554$ $0.7401$ $0.9970$ $0.9985$ $0.9956$ $0.9788$ aKNN $0.7624$ $0.7625$ $0.7625$ $0.7625$ $0.7625$ $0.7625$ $0.7625$ $0.7625$ LOF $0.9719$ $0.9773$ $0.9777$ $0.9779$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ FastABOD $0.7604$ $0.7548$ $0.7495$ $0.7449$ $0.7624$ $0.7625$ $0.7625$ $0.7625$ iForest — — — — $0.7603$ $0.7575$ $0.7450$ $0.7368$
$Pre{c}_{mean}$ $Pre{c}_{max}$ $Method$ / $d$ $10$ $100$ $1000$ $10000$ $10$ $100$ $1000$ $10000$ CFOF $0.9324$ $0.9991$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ ODIN $0.7349$ $0.6012$ $0.5564$ $0.5434$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ antiHub2 $0.6317$ $0.4255$ $0.3346$ $0.3028$ $0.8880$ $0.9560$ $0.9120$ $0.8080$ aKNN $0.4996$ $0.5000$ $0.5000$ $0.5000$ $0.5000$ $0.5000$ $0.5000$ $0.5000$ LOF $0.8743$ $0.9463$ $0.9521$ $0.9521$ $1.0000$ $1.0000$ $1.0000$ $1.0000$ FastABOD $0.4874$ $0.4660$ $0.4506$ $0.4409$ $0.5000$ $0.5000$ $0.5000$ $0.5000$ iForest — — — — $0.4880$ $0.4940$ $0.4680$ $0.4540$

Table 10. Table 10. Accuracy comparison on labelled data.

	$𝐄\left[AU{C}_{max}\right]$	$𝐄\left[AU{C}_{mean}\right]$	$𝐄\left[Pre{c}_{max}\right]$	$𝐄\left[Pre{c}_{mean}\right]$
CFOF	$0.845$	$0.732$	$0.408$	$0.292$
ODIN	$0.784$	$0.717$	$0.394$	$0.253$
aHub2	$0.761$	$0.694$	$0.369$	$0.244$
aKNN	$0.763$	$0.733$	$0.373$	$0.328$
LOF	$0.799$	$0.716$	$0.395$	$0.295$
FastABOD	$0.752$	$0.737$	$0.360$	$0.336$
iForest	$0.760$	—	$0.385$	—
	$p_{win,AU{C}_{max}}$	$p_{win,AU{C}_{mean}}$	$p_{win,Pre{c}_{max}}$	$p_{win,Pre{c}_{mean}}$
CFOF	$0.772$	$0.694$	$0.619$	$0.542$
ODIN	$0.582$	$0.422$	$0.521$	$0.450$
aHub2	$0.353$	$0.237$	$0.553$	$0.392$
aKNN	$0.411$	$0.562$	$0.467$	$0.526$
LOF	$0.565$	$0.399$	$0.612$	$0.554$
FastABOD	$0.414$	$0.686$	$0.395$	$0.576$
iForest	$0.418$	—	$0.454$	—

Table 11. Table 11. Wilcoxon rank-sum test for accuracy ( p 𝑝 p -values).

$p_{rank,AUC}$
	CFOF	ODIN	aHub2	aKNN	LOF	FastABOD	iForest
$CFOF$	—	$0.924$	$0.976$	$0.921$	$0.844$	$0.954$	$0.944$
$ODIN$	$0.076$	—	$0.769$	$0.604$	$0.412$	$0.687$	$0.659$
$aHub2$	$0.024$	$0.231$	—	$0.373$	$0.176$	$0.474$	$0.419$
$aKNN$	$0.079$	$0.396$	$0.627$	—	$0.335$	$0.569$	$0.594$
$LOF$	$0.156$	$0.588$	$0.824$	$0.665$	—	$0.759$	$0.738$
$ABOD$	$0.046$	$0.313$	$0.526$	$0.431$	$0.241$	—	$0.483$
iForest	$0.056$	$0.341$	$0.581$	$0.406$	$0.262$	$0.517$	—

Table 12. Table 12. Maximum A ​ U ​ C 𝐴 𝑈 𝐶 AUC and P ​ r ​ e ​ c 𝑃 𝑟 𝑒 𝑐 Prec values on the labelled datasets.

wdbc dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.827$	$0.855$	$0.849$	$0.744$	$0.802$	$0.476$	$0.842$	$0.263$	$0.272$	$0.119$	$0.013$	$0.200$	$0.032$	$0.116$
$1$	$0.950$	$0.945$	$0.920$	$0.979$	$0.970$	$0.981$	$0.955$	$0.702$	$0.705$	$0.703$	$0.739$	$0.706$	$0.745$	$0.587$
yeast dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.756$	$0.753$	$0.801$	$0.775$	$0.769$	$0.756$	$0.741$	$0.160$	$0.150$	$0.198$	$0.165$	$0.155$	$0.158$	$0.158$
$1$	$0.786$	$0.675$	$0.654$	$0.635$	$0.664$	$0.644$	$0.658$	$0.074$	$0.076$	$0.079$	$0.061$	$0.081$	$0.087$	$0.093$
$2$	$0.809$	$0.749$	$0.734$	$0.786$	$0.751$	$0.798$	$0.746$	$0.162$	$0.161$	$0.161$	$0.139$	$0.155$	$0.161$	$0.119$
$3$	$0.946$	$0.915$	$0.935$	$0.943$	$0.942$	$0.889$	$0.922$	$0.747$	$0.589$	$0.747$	$0.723$	$0.723$	$0.474$	$0.661$
$4$	$0.952$	$0.923$	$0.940$	$0.958$	$0.957$	$0.924$	$0.924$	$0.720$	$0.574$	$0.701$	$0.732$	$0.729$	$0.681$	$0.674$
$5$	$0.794$	$0.772$	$0.786$	$0.779$	$0.814$	$0.731$	$0.757$	$0.385$	$0.333$	$0.355$	$0.371$	$0.377$	$0.284$	$0.329$
$6$	$0.878$	$0.893$	$0.878$	$0.868$	$0.877$	$0.875$	$0.879$	$0.345$	$0.336$	$0.316$	$0.303$	$0.310$	$0.306$	$0.348$
$7$	$0.853$	$0.736$	$0.690$	$0.728$	$0.781$	$0.707$	$0.711$	$0.415$	$0.420$	$0.416$	$0.381$	$0.432$	$0.374$	$0.358$
$8$	$0.881$	$0.789$	$0.758$	$0.728$	$0.809$	$0.735$	$0.731$	$0.538$	$0.540$	$0.494$	$0.352$	$0.555$	$0.419$	$0.419$
$9$	$0.928$	$0.861$	$0.845$	$0.862$	$0.869$	$0.851$	$0.770$	$0.733$	$0.717$	$0.699$	$0.703$	$0.706$	$0.655$	$0.565$
pima-indians-diabetes dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.770$	$0.730$	$0.723$	$0.706$	$0.712$	$0.711$	$0.746$	$0.074$	$0.071$	$0.118$	$0.090$	$0.084$	$0.090$	$0.123$
$1$	$0.749$	$0.651$	$0.612$	$0.562$	$0.647$	$0.561$	$0.555$	$0.066$	$0.065$	$0.081$	$0.029$	$0.129$	$0.029$	$0.039$
segment dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$1$	$0.992$	$0.991$	$0.980$	$0.998$	$0.995$	$0.998$	$0.998$	$0.841$	$0.852$	$0.752$	$0.900$	$0.923$	$0.890$	$0.906$
$2$	$1.000$	$1.000$	$1.000$	$1.000$	$1.000$	$0.994$	$0.999$	$1.000$	$1.000$	$0.990$	$1.000$	$1.000$	$0.984$	$0.913$
$3$	$0.954$	$0.961$	$0.904$	$0.924$	$0.952$	$0.936$	$0.916$	$0.611$	$0.605$	$0.474$	$0.058$	$0.568$	$0.171$	$0.319$
$4$	$0.938$	$0.942$	$0.921$	$0.960$	$0.926$	$0.962$	$0.924$	$0.458$	$0.467$	$0.426$	$0.410$	$0.416$	$0.416$	$0.619$
$5$	$0.946$	$0.940$	$0.879$	$0.962$	$0.940$	$0.964$	$0.938$	$0.582$	$0.595$	$0.529$	$0.613$	$0.542$	$0.635$	$0.590$
$6$	$0.995$	$0.997$	$0.980$	$0.999$	$0.994$	$0.999$	$0.992$	$0.802$	$0.829$	$0.572$	$0.910$	$0.845$	$0.929$	$0.881$
$7$	$1.000$	$1.000$	$0.999$	$1.000$	$1.000$	$0.999$	$0.998$	$0.992$	$0.994$	$0.923$	$0.997$	$0.997$	$0.955$	$0.890$
biodeg dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.824$	$0.839$	$0.810$	$0.843$	$0.809$	$0.849$	$0.871$	$0.253$	$0.257$	$0.192$	$0.165$	$0.181$	$0.197$	$0.326$
$1$	$0.710$	$0.549$	$0.571$	$0.546$	$0.590$	$0.556$	$0.443$	$0.006$	$0.012$	$0.016$	$0.013$	$0.010$	$0.006$	$0.003$
eighthr dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.749$	$0.648$	$0.610$	$0.428$	$0.619$	$0.461$	$0.493$	$0.050$	$0.047$	$0.013$	$0.000$	$0.065$	$0.019$	$0.000$
$1$	$0.813$	$0.739$	$0.678$	$0.731$	$0.712$	$0.746$	$0.820$	$0.260$	$0.240$	$0.274$	$0.281$	$0.255$	$0.297$	$0.452$
wine dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.934$	$0.916$	$0.925$	$0.937$	$0.929$	$0.820$	$0.991$	$0.669$	$0.580$	$0.653$	$0.581$	$0.600$	$0.503$	$0.868$
$1$	$0.818$	$0.801$	$0.799$	$0.851$	$0.836$	$0.823$	$0.894$	$0.545$	$0.457$	$0.539$	$0.545$	$0.542$	$0.381$	$0.503$
$2$	$0.873$	$0.874$	$0.859$	$0.843$	$0.848$	$0.849$	$0.981$	$0.599$	$0.610$	$0.601$	$0.581$	$0.581$	$0.548$	$0.832$
bupa dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.792$	$0.640$	$0.626$	$0.615$	$0.640$	$0.619$	$0.622$	$0.110$	$0.100$	$0.125$	$0.103$	$0.110$	$0.087$	$0.097$
$1$	$0.754$	$0.673$	$0.602$	$0.504$	$0.632$	$0.547$	$0.489$	$0.075$	$0.074$	$0.083$	$0.019$	$0.074$	$0.013$	$0.052$
ionosphere dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.935$	$0.926$	$0.906$	$0.967$	$0.938$	$0.961$	$0.920$	$0.571$	$0.549$	$0.456$	$0.742$	$0.535$	$0.703$	$0.542$
$1$	$0.477$	$0.401$	$0.552$	$0.339$	$0.381$	$0.272$	$0.373$	$0.010$	$0.016$	$0.096$	$0.006$	$0.029$	$0.000$	$0.016$
epileptic dataset
	$AUC$							$Prec$
	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest	CFOF	ODIN	aHub2	aKNN	LOF	ABOD	iForest
$0$	$0.577$	$0.103$	$0.051$	$0.051$	$0.453$	$0.088$	$0.026$	$0.000$	$0.001$	$0.000$	$0.000$	$0.000$	$0.000$	$0.000$
$1$	$0.897$	$0.798$	$0.559$	$0.748$	$0.835$	$0.791$	$0.611$	$0.292$	$0.281$	$0.108$	$0.087$	$0.232$	$0.152$	$0.074$
$2$	$0.902$	$0.800$	$0.623$	$0.750$	$0.849$	$0.804$	$0.638$	$0.229$	$0.227$	$0.200$	$0.226$	$0.261$	$0.232$	$0.219$
$3$	$0.678$	$0.502$	$0.527$	$0.493$	$0.591$	$0.538$	$0.475$	$0.252$	$0.249$	$0.197$	$0.232$	$0.245$	$0.248$	$0.203$
$4$	$0.813$	$0.797$	$0.772$	$0.797$	$0.798$	$0.793$	$0.784$	$0.413$	$0.410$	$0.287$	$0.419$	$0.426$	$0.416$	$0.332$

Equations

$${\mathrm{N}}_k(x)=|\{y:x\in {\mathrm{NN}}_k(y)\}|,$$

$$\text{mbox}CFOF(x)=\underset{1\le k^{\prime }\le n}{\min }\left\{\frac{k^{\prime }}{n}:{\mathrm{N}}_{k^{\prime }}(x)\ge n\varrho \right\}.$$

$${\mu }_k({\mathbf{X}}^{(d)}{)}^h\equiv {\tilde{\mu }}_k^h({\mathbf{X}}^{(d)})=\frac{1}{d}\sum _{i=1}^d\mathbf{E}[(X_i-{\mu }_{X_i}{)}^k{]}^h$$

$$z_{x,{\mathbf{X}}^{(d)}}=\frac{\Vert x^{(d)}-{\mu }_X^{(d)}{\Vert }^2-{\mu }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X^{(d)}{\Vert }^2}}{{\sigma }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X^{(d)}{\Vert }^2}},$$

$${\kappa }_X=\mathbf{E}\left[{\left(\frac{X-{\mu }_X}{{\sigma }_X}\right)}^4\right]=\frac{{\mu }_4(X)}{{\mu }_2(X{)}^2}.$$

$$\text{mbox}CFOF(x^{(d)})$$

$$\forall \kappa >1,Pr[\text{mbox}CFOF({\mathbf{X}}^{(d)})\le s]$$

$${\mathrm{N}}_k(x^{(d)})=n\cdot Pr[x^{(d)}\in {\mathrm{NN}}_k({\mathbf{X}}^{(d)})]\approx n\Phi \left(\frac{{\Phi }^{-1}(\frac{k}{n})\sqrt{{\mu }_4+3{\mu }_2^2}-z_x\sqrt{{\mu }_4-{\mu }_2^2}}{2{\mu }_2}\right).$$

$$\tau (z_x)\approx n\Phi \left(\frac{z_x\sqrt{{\mu }_4-{\mu }_2^2}+2{\mu }_2{\Phi }^{-1}(\varrho )}{\sqrt{{\mu }_4+3{\mu }_2^2}}\right).$$

$$\begin{gathered} \text{mbox}CFOF(x^{(d)}\le s)⟺\Phi \left(\frac{z_x\sqrt{\kappa -1}+2{\Phi }^{-1}(\varrho )}{\sqrt{\kappa +3}}\right)\le s⟺ \\ ⟺z_x\le \frac{{\Phi }^{-1}(s)\sqrt{\kappa +3}-2{\Phi }^{-1}(\varrho )}{\sqrt{\kappa -1}}⟺z_x\le t(s). \end{gathered}$$

$$Pr\left[\frac{\Vert {\mathbf{X}}^{(d)}-{\mu }_X{\Vert }^2-{\mu }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X{\Vert }^2}}{{\sigma }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X{\Vert }^2}}\le z\right]=\Phi (z).$$

$$f_{\mathrm{CFOF}}(s)=\frac{\mathrm{d}}{\mathrm{d}s}{F}_{\mathrm{CFOF}}(s)=\sqrt{\frac{\kappa +3}{\kappa -1}}\cdot \frac{1}{\varphi ({\Phi }^{-1}(s))}\cdot \varphi \left(\frac{{\Phi }^{-1}(s)\sqrt{\kappa +3}-2{\Phi }^{-1}(\varrho )}{\sqrt{\kappa -1}}\right).$$

$$\underset{d\to \infty }{\lim }Pr\left[\left|\frac{s{c}_{Def}({\mathbf{X}}^{(d)})-s{c}_{Def}({x_0}^{(d)})}{s{c}_{Def}({x_0}^{(d)})}\right|\le ϵ\right]=1,$$

$$\underset{d\to \infty }{\lim }\Phi (z_{{x_1}^{(d)}})\to 0\text{mbox}and\underset{d\to \infty }{\lim }\Phi (z_{{x_2}^{(d)}})\to 1,$$

$$s{c}_{Def}({x_1}^{(d)})=(1-ϵ)s{c}_{Def}(x^{(d)})\text{mbox}ands{c}_{Def}({x_2}^{(d)})=(1+ϵ)s{c}_{Def}(x^{(d)}).$$

$$Pr\left[s{c}_{Def}({\mathbf{X}}^{(d)})\le s{c}_{Def}(x^{(d)})\right]=\Phi (z_x).$$

$$\begin{gathered} \underset{d\to \infty }{\lim }Pr\left[\left|\frac{s{c}_{Def}({\mathbf{X}}^{(d)})-s{c}_{Def}({x_0}^{(d)})}{s{c}_{Def}({x_0}^{(d)})}\right|\le ϵ\right]= \\ =\underset{d\to \infty }{\lim }Pr\left[(1-ϵ)s{c}_{Def}({x_0}^{(d)})\le s{c}_{Def}({\mathbf{X}}^{(d)})\le (1+ϵ)s{c}_{Def}({x_0}^{(d)})\right]=1, \end{gathered}$$

$$\underset{d\to \infty }{\lim }\left(\Phi (z_{{x_2}^{(d)}})-\Phi (z_{{x_1}^{(d)}})\right)=1,$$

$$z_{x_1}=\frac{{\Phi }^{-1}\left((1-ϵ)\text{mbox}CFOF(x_0^{(d)})\right)\sqrt{\kappa +3}-2{\Phi }^{-1}(\varrho )}{\sqrt{\kappa -1}}.$$

$$\underset{{\kappa }_X\to 1^{+}}{\lim }\text{mbox}CFOF(x^{(d)})$$

$$\underset{{\kappa }_X\to +\infty }{\lim }\text{mbox}CFOF(x^{(d)})$$

$$s{c}_{Def}(x^{(d)},{\mathbf{X}}^{(d)})=s{c}_{Def}(a{x}^{(d)}+b^{(d)},{\mathbf{Y}}^{(d)})$$

$$z_{y,{\mathbf{Y}}^{(d)}}=\frac{\Vert y^{(d)}-{\mu }_Y{\Vert }^2-{\mu }_{\Vert {\mathbf{Y}}^{(d)}-{\mu }_Y{\Vert }^2}}{{\sigma }_{\Vert {\mathbf{Y}}^{(d)}-{\mu }_Y{\Vert }^2}}=\frac{a^2\Vert x^{(d)}-{\mu }_X{\Vert }^2-a^2{\mu }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X{\Vert }^2}}{a^2{\sigma }_{\Vert {\mathbf{X}}^{(d)}-{\mu }_X{\Vert }^2}}=z_{x,{\mathbf{X}}^{(d)}}.$$

$$\mbox{CFOF}(a{x}^{(d)}+{b}^{(d)},{\bm{Y}^{(d)}})=\Phi\left(\frac{z_{y}\sqrt{\mu_{4}(Y)-\mu_{2}(Y)^{2}}+2\mu_{2}(Y)\Phi^{-1}(\varrho)}{\sqrt{\mu_{4}(Y)+3\mu_{2}(Y)^{2}}}\right)=\\ =\Phi\left(\frac{z_{x}\sqrt{a^{4}\mu_{4}(X)-\big{(}a^{2}\mu_{2}(X)\big{)}^{2}}+2a^{2}\mu_{2}(X)\Phi^{-1}(\varrho)}{\sqrt{a^{4}\mu_{4}(X)+3\big{(}a^{2}\mu_{2}(X)\big{)}^{2}}}\right)=\\ =\Phi\left(\frac{z_{x}\sqrt{\mu_{4}(X)-\mu_{2}(X)^{2}}+2\mu_{2}(X)\Phi^{-1}(\varrho)}{\sqrt{\mu_{4}(X)+3\mu_{2}(X)^{2}}}\right)=\mbox{CFOF}({x}^{(d)},{\bm{X}^{(d)}}).$$

$$\frac{{\alpha }_i}{{\pi }_i}=1-F_{\mathrm{CFOF}}\left(\frac{s^{\ast }}{{\pi }_i};\frac{\varrho }{{\pi }_i},{{\mathbf{Y}}_{\mathbf{i}}}^{(d)}\right),$$

$$F_{\mathrm{CFOF}}(s;\varrho ,{\mathbf{Y}}^{(d)})=\sum _{i=1}^K{\pi }_i{F}_{\mathrm{CFOF}}\left(\frac{s}{{\pi }_i};\frac{\varrho }{{\pi }_i},{{\mathbf{Y}}_{\mathbf{i}}}^{(d)}\right).$$

$$F_{\mathrm{CFOF}}(s;\varrho ,{\mathbf{Y}}^{(d)})=K\cdot F_{\mathrm{CFOF}}\left(sK;\varrho K,{{\mathbf{Y}}_0}^{(d)}\right),$$

$$\begin{gathered} \text{mbox}KNN(x^{(d)})=\mathrm{dist}\left(x^{(d)},{nn}_k(x^{(d)})\right)\approx \sqrt{\Vert x^{(d)}{\Vert }^2+{\mu }_{\Vert {\mathbf{X}}^{(d)}{\Vert }^2}+{\Phi }^{-1}\left(k/n\right){\sigma }_{\Vert x^{(d)}-{\mathbf{X}}^{(d)}{\Vert }^2}}= \\ =\sqrt{\Vert x^{(d)}{\Vert }^2+d{\mu }_2+{\Phi }^{-1}\left(k/n\right)\sqrt{d({\mu }_4-{\mu }_2^2)+4{\mu }_2\Vert x^{(d)}{\Vert }^2}}. \end{gathered}$$

$$\text{mbox}KNN(x^{(d)})\approx \sqrt{\Vert x^{(d)}{\Vert }^2+d{\mu }_2}.$$

$$\text{mbox}KNN(x^{(d)})\approx \sqrt{z_x\sqrt{d({\mu }_4-{\mu }_2^2)}+2d{\mu }_2}.$$

$$\text{mbox}KNN({x_1}^{(d)})=$$

Full text

Outlier detection, high-dimensional data, curse of dimensionality, concentration of distances, hubness, kurtosis, huge datasets

††journal: TKDD††journalvolume: V††journalnumber: N††article: 1††journalyear: 2019††publicationmonth: 9††ccs: Computing methodologies Unsupervised learning††ccs: Computing methodologies Anomaly detection

1. Introduction

Outlier detection is one of the main data mining and machine learning tasks, whose goal is to single out anomalous observations, also called outliers (Aggarwal, 2013). While the other data analysis approaches, such as classification, clustering or dependency detection, consider outliers as noise that must be eliminated, as pointed out in (Han and Kamber, 2001), “one person’s noise could be another person’s signal”, thus outliers themselves are of great interest in different settings, e.g. fraud detection, ecosystem disturbances, intrusion detection, cybersecurity, medical analysis, to cite a few.

Outlier analysis has its roots in statistics (Davies and Gather, 1993; Barnett and Lewis, 1994). Data mining outlier approaches to outlier detection can be classified in supervised, semi-supervised, and unsupervised (Hodge and Austin, 2004; Chandola et al., 2009). Supervised methods take in input data labeled as normal and abnormal and build a classifier. The challenge there is posed by the fact that abnormal data form a rare class. Semi-supervised methods, also called one-class classifiers or domain description techniques, take in input only normal examples and use them to identify anomalies. Unsupervised methods detect outliers in an input dataset by assigning a score or anomaly degree to each object.

A commonly accepted definition fitting the unsupervised setting is the following: “Given a set of data points or objects, find those objects that are considerably dissimilar, exceptional or inconsistent with respect to the remaining data” (Han and Kamber, 2001).

Unsupervised outlier detection methods can be categorized in several approaches, each of which assumes a specific concept of outlier. Among the most popular families there are statistical-based (Davies and Gather, 1993; Barnett and Lewis, 1994), deviation-based (Arning et al., 1996), distance-based (Knorr and Ng, 1998; Ramaswamy et al., 2000; Angiulli and Pizzuti, 2005; Angiulli and Fassetti, 2009), density-based (Breunig et al., 2000; Jin et al., 2001; Papadimitriou et al., 2003; Jin et al., 2006), reverse nearest neighbor-based (Hautamäki et al., 2004; Radovanović et al., 2015) angle-based (Kriegel et al., 2008), isolation-based (Liu et al., 2012), subspace-based (Aggarwal and Yu, 2001; Angiulli et al., 2009; Keller et al., 2012), ensemble-based (Lazarevic and Kumar, 2005; Aggarwal and Sathe, 2017) and others (Chandola et al., 2012; Zimek et al., 2012; Aggarwal, 2013; Akoglu et al., 2015).

This work focuses on unsupervised outlier detection in the full feature space. In particular, we present a novel notion of outlier, the Concentration Free Outlier Factor (CFOF), having the peculiarity to resist to the distance concentration phenomenon which is part of the so called curse of dimensionality problem (Bellman, 1961; Demartines, 1994; Beyer et al., 1999; Chávez et al., 2001; Francois et al., 2007; Angiulli, 2018). Specifically, the term distance concentration refers to the tendency of distances to become almost indiscernible as dimensionality increases. This phenomenon may greatly affect the quality and performances of data mining, machine learning, and information retrieval techniques, since all these techniques rely on the concept of distance, or dissimilarity, among data items in order to retrieve or analyze information. Whereas low-dimensional spaces show good agreement between geometric proximity and the notion of similarity, as dimensionality increases, counterintuitive phenomena like distance concentration and hubness may be harmful to traditional techniques. In fact, the concentration problem also affects outlier scores of different families due to the specific role played by distances in their formulation.

This characteristics of high dimensional data has generated in data analysis and data management applications the need for dimensionality resistant notions of similarity, that are similarities not affected by the poor separation between the furthest and the nearest neighbor in high dimensional space. Among the desiderata that a good distance resistant to dimensionality should possess, there are the to be contrasting and statistically sensitive, that is meaningfully refractory to concentration, and to be compact, or efficiently computable in terms of time and space (Aggarwal, 2001).

In the context of unsupervised outlier detection, (Zimek et al., 2012) identified different issues related to the treatment of high-dimensional data, among which the concentration of scores, in that derived outlier score become numerically similar, interpretability of scores, that fact that the scores often no longer convey a semantic meaning, and hubness, the fact that certain points occur more frequently in neighbor lists than others (Aucouturier and Pachet, 2008; Radovanović et al., 2009; Angiulli, 2018).

Specifically, consider the number ${\rm N}_{k}(x)$ of observed points that have $x$ among their $k$ nearest neighbors, also called $k$-occurrences or reverse $k$-nearest neighbor count, or RNNc, for short, in the following. It is known that in low dimensional spaces, the distribution of ${\rm N}_{k}(x)$ over all $x$ complies with the binomial distribution and, in particular, for uniformly i.i.d. data in low dimensions, that it can be modeled as node in-degrees in the $k$-nearest neighbor graph, which follows the Erdős-Rényi graph model (Erdös and Rényi, 1959). However, it has been observed that as the dimensionality increases, the distribution of ${\rm N}_{k}$ becomes skewed to the right, resulting in the emergence of hubs, which are points whose reverse nearest neighbors counts tend to be meaningfully larger than that associated with any other point.

Thus, the circumstance that the outlier scores tend to be similar poses some challenges in terms of their intelligibility, absence of a clear separation between outliers and inliers, and loss of efficiency of pruning rules aiming at reducing the computational effort.

The CFOF score is a reverse nearest neighbor-based score. Loosely speaking, it corresponds to measure how many nearest neighbors have to be taken into account in order for a point to be close to a sufficient fraction $\varrho$ of the data population. We notice that this kind of notion of perceiving the abnormality of an observation is completely different from any other notion so far introduced. In the literature, there are other outlier detection approaches resorting to reverse nearest neighbor counts (Hautamäki et al., 2004; Jin et al., 2006; Lin et al., 2008; Radovanović et al., 2015). Methods such INFLO (Jin et al., 2006) are density-based techniques considering both direct and reverse nearest neighbors when estimating the outlierness of a point. Early reverse nearest neighbor-based approaches, that are ODIN (Hautamäki et al., 2004), which uses ${\rm N}_{k}(x)$ as outlier score of $x$, and the one proposed in (Lin et al., 2008), which returns as outliers those points $x$ having ${\rm N}_{k}(x)=0$, are prone to the hubness phenomenon, that is the concentration of the scores towards the values associated with outliers, due to direct used of the function ${\rm N}_{k}$. Hence, to mitigate the hubness effect, (Radovanović et al., 2015) proposed a simple heuristic method, namely AntiHub2, which refines the scores produced by the ODIN method by returning the weighted mean of the sum of the ${\rm N}_{k}$ scores of the neighbors of the point and of the ${\rm N}_{k}$ score of the point itself.

In this work we both empirically and theoretically show that the here introduced CFOF outlier score complies with all of the above mentioned desiderata. As a main contribution, we formalize the notion of concentration of outlier scores, and theoretically prove that CFOF does not concentrate in the Euclidean space $\mathbb{R}^{d}$ for any arbitrarily large dimensionality $d\rightarrow\infty$. To the best of our knowledge, there are no other proposals of outlier detection measures, and probably also of other data analysis measures related to the Euclidean distance, for which it has been provided the theoretical evidence that they are immune to the concentration effect.

We recognize that the kurtosis $\kappa$ of the data population, a well-known measure of tailedness of a probability distribution originating with Karl Pearson (Pearson, 1905; Fiori and Zenga, 2009; Westfall, 2014), is a key parameter for characterizing from the outlier detection perspective the unknown distribution underlying the data, a fact that has been neglected at least within the data mining literature. The kurtosis may range from $\kappa=1$, for platykurtic distributions such as the Bernoulli distribution with success probability $0.5$, to $\kappa\rightarrow\infty$, for extreme leptokurtic or heavy-tailed distributions. Each outlier score must concentrate for $\kappa=1$ due to the absolute absence of outliers. We prove that CFOF does not concentrate for any $\kappa>1$.

We determine the closed form of the distribution of the CFOF scores for arbitrarily large dimensionalities and show that the CFOF score of a point $x$ depends, other than on the parameter $\varrho$ employed, on its squared norm standard score $z_{x}$ and on the kurtosis $\kappa$ of the data distribution. The squared norm standard score of a data point is the standardized squared norm of the point under the assumption that the origin of the feature space coincides with the mean of the distribution generating points. We point out that the knowledge of the theoretical distribution of an outlier score is a rare, if not unique, peculiarity. We prove that the probability to observe larger scores increases with the kurtosis.

As for the hubness phenomenon, by exploiting the closed form of the CFOF scores distribution, we provide evidence that CFOF does not suffer of the hubness problem, since points associated with the largest scores always correspond to a small fraction of the data. Moreover, while previously known RNNc scores present large false positive rates for values of the parameter $k$ which are not comparable with $n$, CFOF is able to establish a clear separation between outliers and inliers for any value of the parameter $\varrho$.

We theoretically prove that the CFOF score is both translation and scale-invariant. This allows to establish that CFOF has connections with local scores. Indeed, if we consider a dataset consisting of multiple translated and scaled copies of the same seed cluster, the set of the CFOF outliers consists of the same points from each cluster. More in the general, in the presence of clusters having different generating distributions, the number of outliers coming from each cluster is directly proportional to its size and to its kurtosis, a property that we called semi–locality.

As an equally important contribution, the design of the novel outlier score and the study of its theoretical properties allowed us to shed lights also on different properties of well-known outlier detection scores.

First, we determine that the semi–locality is a peculiarity of reverse nearest neighbor counts. This discovery clarifies the exact nature of the reverse nearest neighbor family of outlier scores: while in the literature this family of scores has been observed to be adaptive to different density levels, the exact behavior of this adaptivity was unclear till now.

Second, we identify the property each outlier score which is monotone increasing with respect to the squared norm standard score must possess in order to avoid concentration. We leverage this property to formally show that classic distance-based and density-based outlier scores are subject to concentration both for bounded and unbounded dataset sizes $n$, and both for fixed and variable values of the parameter $k$. Moreover, the convergence rate towards concentration of these scores is inversely proportional to the kurtosis of the data.

Third, as a theoretical confirmation of the proneness of ${\rm N}_{k}$ to false positives, we show that the ratio between the amount of variability of the CFOF outlier scores and that of the RNNc outlier scores corresponds to of several orders of magnitude and, moreover, that the above ratio is even increasing with the kurtosis.

Local outlier detection methods, showing adaptivity to different density levels, are usually identified in the literature with those methods that compute the output scores by comparing the neighborhood of each point with the neighborhood of its neighbors. We point out that, as far as CFOF is concerned, its local behavior is obtained without the need to explicitly perform such a kind of comparison. Rather, since from a conceptual point of view computing CFOF scores can be assimilated to estimate a probability, we show that CFOF scores can be reliably computed by exploiting sampling techniques. The reliability of this approach descends from the fact that CFOF outliers are the points less prone to bad estimations.

Specifically, to deal with very large and high-dimensional datasets, we introduce the fast-CFOF technique which exploits sampling to avoid the computation of exact nearest neighbors and, hence, from the computational point of view does not suffer of the dimensionality curse affecting (reverse) nearest neighbor search techniques. The cost of fast-CFOF is linear both in the dataset size and dimensionality. The fast-CFOF algorithm is efficiently parallelizable, and we provide a multi-core (MIMD) vectorized (SIMD) implementation.

The algorithm has an unique parameter $\varrho\in(0,1)$, representing a fraction of the data population. The fast-CFOF algorithm supports multi-resolution analysis regarding the dataset at different scales, since different $\varrho$ values can be managed simultaneously by the algorithm, with no additional computational effort.

Experimental results highlight that fast-CFOF is able to achieve very good accuracy with reduced sample sizes $s$ and, hence, to efficiently process huge datasets. Moreover, since its asymptotic cost does not depend on the actual value of the parameter $\varrho$, CFOF can efficiently manage even large values of this parameter, a property which is considered a challenge for different existing outlier methods. Moreover, experiments involving the CFOF score witness for the absence of concentration on real data, show that CFOF shows excellent accuracy performances on distribution data, and that CFOF is likely to admit configurations ranking the outliers better than other approaches on labelled data.

The study of theoretical properties is conducted by considering the Euclidean distance as dissimilarity measure, but from (Angiulli, 2018) it is expected they are also valid for any Minkowski’s metrics. Moreover, it is worth to notice that the applicability of the technique is not confined to the Euclidean space or to vector spaces. It can be applied both in metric and non-metric spaces equipped with a distance function. Moreover, while effectiveness and efficiency of the method do not deteriorate with the dimensionality, its application is perfectly reasonable even in low dimensions.

We believe the CFOF technique and the properties presented in this work provide insights within the scenario of outlier detection and, more in the general, of high-dimensional data analysis.

The rest of the work is organized as follows. Section 2 introduces the CFOF score and provides empirical evidence of its behavior. Section 3 studies theoretical properties of the CFOF outlier score. Section 4 presents the fast-CFOF algorithm for detecting outliers in large high-dimensional datasets. Section 5 describes experiments involving the propose approach. Finally, Section 6 draws conclusions and depicts future work.

2. The Concentration Free Outlier Factor

In this section, we introduce the Concentration Free Outlier Factor (CFOF), a novel outlier detection measure.

After presenting the definition of CFOF score (see Section 2.1), we provide empirical evidence of its behavior by discussing relationship with the distance concentration phenomenon (see Section 2.2) and with the hubness phenomenon (see Section 2.3). Theoretical properties of CFOF will be taken into account in subsequent Section 3.

2.1. Definition

Let ${\bf DS}=\{x_{1},x_{2},\ldots,x_{n}\}$ denote a dataset of $n$ points, also said objects, belonging to an object space $\mathbb{U}$ equipped with a distance function ${\rm dist}$. In the following, we assume that $\mathbb{U}$ is a vector space of the form $\mathbb{U}=\mathbb{D}^{d}$, where $d\in\mathbb{N}^{+}$, the dimensionality of the space, is a positive natural number and $\mathbb{D}$ is usually the set $\mathbb{R}$ of real numbers. However, we point out that the method can be applied in any object space equipped with a distance function (not necessarily a metric).

Given a dataset object $x$ and a positive integer $k$, the $k$-th nearest neighbor of $x$ is the dataset object ${\it nn}_{k}(x)$ such that there exists exactly $k-1$ dataset objects lying at distance smaller than ${\rm dist}(x,{\it nn}_{k}(x))$ from $x$. It always holds that $x={\it nn}_{1}(x)$. We assume that ties are non-deterministically ordered.

The $k$ nearest neighbors set ${\rm NN}_{k}(x)$ of $x$, where $k$ is also said the neighborhood width, is the set of objects $\{{\it nn}_{i}(x)\mid 1\leq i\leq k\}$.

By ${\rm N}_{k}(x)$ we denote the number of objects having $x$ among their $k$ nearest neighbors:

[TABLE]

also referred to as $k$-occurrences function or reverse neighborhood size or reverse $k$ nearest neighbor count or RNNc, for short.

Definition 2.1 (CFOF outlier score).

Given a parameter $\varrho\in(0,1)$, the Concentration Free Outlier Score, also referred to as CFOF (or $\varrho$–CFOF if the value of the parameter $\varrho$ is not clear from the context), is defined as:

[TABLE]

Thus, the CFOF score of $x$ represents the smallest neighborhood width, normalized with respect to $n$, for which $x$ exhibits a reverse neighborhood of size at least $n\varrho$.

The CFOF score belongs to the interval $[0,1]$. In some cases, we will use absolute CFOF score values, ranging from $1$ to $n$.

For complying with existing outlier detection measures that employ the neighborhood width $k$ as an input parameter, when we refer to the input parameter $k$, we assume that, as far as CFOF is concerned, $k$ represents a shorthand for the parameter $\varrho=k/n$.

Figure 1 illustrates the computation of the CFOF score on a two-dimensional example dataset.

Intuitively, the CFOF score measures how many neighbors have to be taken into account in order for the object to be considered close by an appreciable fraction of the dataset objects. We point out that this kind of notion of perceiving the abnormality of an observation is completely different from any other notion so far introduced in the literature.

In particular, the point of view here is in some sense reversed with respect to distance-based outliers, since we are interested in determining the smallest neighborhood width $k^{\prime}$ for which the object is a neighbor of at least $n\varrho$ other objects, while distance-based outliers (and, specifically, the definition considering the distance from the $k$th nearest neighbor) determine the smallest radius of a region centered in the object which contains at least $k$ other objects.

2.2. Relationship with the distance concentration phenomenon

One of the main peculiarities of the CFOF definition is its resistance to the distance concentration phenomenon, which is part of the so called curse of dimensionality problem (Demartines, 1994; Beyer et al., 1999; Francois et al., 2007; Angiulli, 2018). As already recalled, the term curse of dimensionality is used to refer to difficulties arising when high-dimensional data must be taken into account, and one of the main aspects of this curse is distance concentration, that is the tendency of distances to become almost indiscernible as dimensionality increases.

In this scenario (Demartines, 1994) has shown that the expectation of the Euclidean distance of i.i.d. random vectors increases as the square root of the dimension, whereas its variance tends toward a constant. This implies that high-dimensional points appear to be distributed around the surface of a sphere and distances between pairs of points tend to be similar: according to (Angiulli, 2018), the expected squared Euclidean distance of the points from their mean is $d\sigma^{2}$ and the expected squared Euclidean inter-point distance is $2d\sigma^{2}$, where $\sigma$ is the standard deviation of the random variable used to generate points.

The distance concentration phenomenon is usually characterized in the literature by means of a ratio between some measure related to the spread and some measure related to the magnitude of the distances. In particular, the conclusion is that there is concentration when the above ratio converges to [math] as the dimensionality tends to infinity. The relative variance $RV=\sigma_{dist}/\mu_{dist}$ (Francois et al., 2007) is a measure of concentration for distributions, corresponding to the ratio between the standard deviation $\sigma_{dist}$ and the expected value $\mu_{dist}$ of the distance between pairs of points. In the case of the Euclidean distance, or of any other Minkowski’s metric, the relative variance of data points generated by i.i.d. random vectors tends to zero as the dimensionality tends to infinity, independently from the sample size. As a consequence, the separation between the nearest neighbor and the farthest neighbor of a given point tend to become increasingly indistinct as the dimensionality increases.

Due to the specific role played by distances in their formulation, the concentration problem also affects outlier scores.

To illustrate, Figure 2 shows the ratio ${\sigma_{sc}}/{\mu_{sc}}$ between the standard deviation $\sigma_{sc}$ and the mean $\mu_{sc}$ of different families of outlier scores $sc$, that are the distance-based method aKNN (Angiulli and Pizzuti, 2005), the density-based method LOF (Breunig et al., 2000), the angle-based method ABOF (Kriegel et al., 2008), and CFOF, associated with a family of uniformly distributed (Figure 2a) and a normally distributed (Figure 2b) datasets having fixed size $n=1,\!000$ and increasing dimensionality $d\in[10^{0},10^{4}]$, for $k=50$. We also computed the ratio ${\sigma_{sc}}/{\mu_{sc}}$ for the iForest (Liu et al., 2012) outlier score $sc$, obtaining the decreasing sequence $0.103$, $0.078$, $0.032$, $0.023$, and $0.018$ for $d\in[10^{0},10^{4}]$ on normally distributed data and similar values on uniform data.111The values for the parameters of the iForest algorithm employed throughout the paper correspond to the default values suggested in (Liu et al., 2012), that is to say the sub-sampling size $\psi=256$ and the number of isolation trees $t=100$.

Results for each dimensionality value $d$ are obtained by ($i$) considering ten randomly generated different datasets, ($ii$) computing outlier scores associated with each dataset, ($iii$) sorting scores of each dataset, and ($iv$) taking the average value for each rank position. The figure and the values above reported highlight that, except for CFOF, the other scores, belonging to three different families of techniques, exhibit a concentration effect.

Figure 3 reports the sorted scores of the uniform datasets above discussed. For aKNN (Figure 3a) the mean score value raises while the spread stay limited. For LOF (Figure 3b) all the values tend to $1$ as the dimensionality increases. For ABOF (Figure 3c) both the mean and the standard deviation decrease of various orders of magnitude with the latter term varying at a faster rate than the former one. As for iForest (Figure 3d) the mean stay fixed while the spread decreases. As for CFOF (Figure 3e) the score distributions for $d>100$ are very close and exhibit only small differences.

In order to visualize score distributions, Figure 4 report the scatter plot of the score values versus the distance from the mean for $d=10^{4}$. As for iForest is concerned, we reported both the scatter plot for $d=10$ (Figure 4d) that the scatter plot for $d=10,\!000$ (Figure 4e). Indeed, while iForest associates larger score values to the points maximizing their distance from the mean for small dimensionalities (e.g. $d=10$), when the dimensionality increases the quality of the scores worsens. The scatter plot in Figure 4e ($d=10,\!000$) shows that the iForest score is almost independent of the distance from the mean, thus witnessing that its discrimination capability is practically lost. This can be justified by the kind of strategy pursued by iForest and by the geometry of intrinsically high-dimensional spaces. Since the strategy of iForest consists in recursively partitioning the space by means of randomly generated axis-parallel hyperplanes and then using as a score the recursion depth at which a particular point becomes isolated, it is conceivable that this procedure is destined to output about the same values if the data points are distributed around the surface of an hypersphere, as they tend to do in the case of intrinsically high-dimensional spaces.

2.3. Relationship with the hubness phenomenon

It descends from its definition that CFOF has connections with the reverse neighborhood size, a tool which has been also used for characterizing outliers. The ODIN method (Hautamäki et al., 2004) uses of the reverse neighborhood size ${\rm N}_{k}(\cdot)$ as an outlier score, which we refer also as RNN count, or RNNc for short. Outliers are those objects associated with the smallest RNN counts. However, it is well-known that the function ${\rm N}_{k}(\cdot)$ suffers of a peculiar problem known as hubness (Aucouturier and Pachet, 2008; Radovanović et al., 2009; Angiulli, 2018). As the dimensionality of the space increases, the distribution of $N_{k}(\cdot)$ becomes skewed to the right with increasing variance, leading to a very large number of objects showing very small RNN counts. Thus, the number of antihubs, that are objects appearing in a much smaller number of $k$ nearest neighbors sets (possibly they are neighbors only of themselves), overcomes the number of hubs, that are objects that appear in many more $k$ nearest neighbor sets than other points, and, according to the RNNc score, the vast majority of the dataset objects become outliers with identical scores.

Here we provide empirical evidence that CFOF does not suffer of the hubness problem, while we refer the reader to Section 3 for the formal demonstration of this property.

Figures 5a and 5b report the distribution of the $N_{k}(\cdot)$ value and of the CFOF absolute score for a ten thousand dimensional uniform dataset. Notice that CFOF outliers are associated with the largest score values, hence to the tails of the distribution, while RNNc outliers are associated with the smallest score values, hence with the largely populated region of the score distribution, a completely opposite behavior.

To illustrate the impact of the hubness problem with the dimensionality, Figures 5c and 5d show the cumulated frequency associated with the normalized, between [math] and $1$, increasing scores. The normalization has been implemented to ease comparison. As for CFOF, values have been obtained as $\frac{{\rm CFOF}(x)}{\max_{y}{\rm CFOF}(y)}$. As for RNNc, values have been obtained as $1-\frac{{\rm N}_{k}(x)}{\max_{y}{\rm N}_{k}(y)}$.

The curves clarify the deep difference between the two approaches. Here both $n$ and $k$ are held fixed, while $d$ is increasing ($d\in\{10^{0},10^{1},10^{2},10^{4}\}$, the curve for $d=10^{3}$ is omitted for readability, since it is very close to $10^{4}$). As for RNNc, the hubness problem is already evident for $d=10$, where objects with a normalized score $\geq 0.8$ correspond to about the $20\%$ of the dataset, while the curve for $d=10^{2}$ closely resembles that for $d=10^{4}$, where the vast majority of the dataset objects have a normalized score close to $1.0$. As for CFOF, the number of points associated with large score values always corresponds to a very small fraction of the dataset population.

3. Concentration free property of CFOF

In this section we theoretically ascertain properties of the CFOF outlier score.

The rest of the section is organized as follows. Section 3.1 introduces the notation exploited throughout the section and some basic definitions. Section 3.2 recalls the concept of kurtosis. Section 3.3 derives the theoretical cumulative distribution function of the CFOF score. Section 3.4 provides the definition of concentration of outlier scores together with the proof that the CFOF score does not concentrate and Section 3.5 discusses the effect of the data kurtosis on the distribution of CFOF outlier scores. Section 3.6 studies the behavior CFOF in presence of different distributions and establishes its semi–locality property. Finally, Section 3.7 studies the concentration properties of distance-based and density-based outlier scores, and Section 3.8 studies the concentration properties of reverse nearest neighbor-based outlier scores.

3.1. Preliminaries

The concept of intrinsic dimensionality is related to the analysis of independent and identically distributed (i.i.d.) data. Although variables used to identify each datum could not be statistically independent, ultimately, the intrinsic dimensionality of the data is identified as the minimum number $D$ of variables needed to represent the data itself (van der Maaten et al., 2009). This corresponds in linear spaces to the number of linearly independent vectors needed to describe each point. Indeed, if random vector components are not independent, the concentration phenomenon is still present provided that the actual number $D$ of “degrees of freedom” is sufficiently large (Demartines, 1994). Thus, results derived for i.i.d. data continue to be valid provided that the dimensionality $d$ is replaced with $D$.

In the following we use $d$-dimensional i.i.d. random vectors as a model of intrinsically high dimensional space. Specifically, boldface uppercase letters with the symbol “$(d)$” as a superscript, such as ${\bm{X}^{(d)}}$, ${\bm{Y}^{(d)}}$, $\ldots$, denote $d$-dimensional random vectors taking values in $\mathbb{R}^{d}$. The components $X_{i}$ ($1\leq i\leq d$) of a random vector ${\bm{X}^{(d)}}=(X_{1},X_{2},\ldots,X_{d})$ are random variables having pdfs $f_{X_{i}}$ (cdf $F_{X_{i}}$). A random vector is said independent and identically distributed (i.i.d.), if its components are independent random variables having common pdf $f_{X}$ (cdf $F_{X}$). The generic component $X_{i}$ of ${\bm{X}^{(d)}}$ is also referred to as $X$ when its position does not matter.

Lowercase letters with “$(d)$” as a superscript, such as ${x}^{(d)}$, ${y}^{(d)}$, $\ldots$, denote a specific $d$-dimensional vector taking value in $\mathbb{R}^{d}$.

Given a random variable $X$, $\mu_{X}$ and $\sigma_{X}$ denote the mean and standard deviation of $X$. The symbol $\mu_{k}(X)$, or simply $\mu_{k}$ when $X$ is clear from the context, denotes the $k$th central moment $\mathbf{E}[(X-\mu_{X})^{k}]$ of $X$ ($k\in\mathbb{N}$). When we used moments, we assume that they exists finite.

A sequence of independent non-identically distributed random variables $X_{1},X_{2},\ldots,X_{d}$ having non-null variances and finite central moments $\mu_{i,k}$ ($1\leq i\leq d$) is said to have comparable central moments if there exist positive constants $\mu_{\max}\geq\max\{|\mu_{i,k}|\}$ and $\mu_{\min}\leq\min\{|\mu_{i,k}|:\mu_{i,k}\neq 0\}$. Intuitively, this guarantees that the ratio between the greatest and the smallset non-null moment remains limited.

An independent non-identically distributed random vector ${\bm{X}^{(d)}}$ whose components have comparable central moments, can be treated as an i.i.d. random vector whose generic component $X_{i}$ is such that the $h$-th degree ($h\in\mathbb{N}$) of its $k$th central moment $\mu_{k}(X_{i})$ ($k\in\mathbb{N}$), that is $\mu_{k}(X_{i})^{h}$, is given by the average of the $h$-th degree of the central moments of its components $X_{i}$ (Angiulli, 2018), defined as follows

[TABLE]

This means that all the results given for i.i.d. $d$-dimensional random vectors can be immediately extended to $d$-dimensional independent non-identically distributed random vectors having comparable central moments and, more in the general, to the wider class of real-life data having $D$ degrees of freedom with comparable central moments.

$\Phi(\cdot)$ ($\phi(\cdot)$, resp.) denotes the cdf (pdf, resp.) of the normal standard distribution.

Results given for distributions in the following, can be applied to finite set of points by assuming large samples.

Definition 3.1 (Squared norm standard score).

Let ${x}^{(d)}$ be a realization of a random vector ${\bm{X}^{(d)}}$. Then, $z_{x,{\bm{X}^{(d)}}}$ denotes the squared norm standard score of ${x}^{(d)}$, that is

[TABLE]

where ${\mu}^{(d)}_{X}=(\mu_{X_{1}},\mu_{X_{2}},\ldots,\mu_{X_{d}})$ denotes the mean vector of ${\bm{X}^{(d)}}$. Clearly, if all the components of the mean vector ${\mu}^{(d)}_{X}$ assume the same value $\mu_{X}$, e.g. as in the case of i.i.d. vectors, then ${\mu}^{(d)}_{X}$ can be replaced by $\mu_{X}$. The notation $z_{x}$ is used as a shorthand for $z_{x,{\bm{X}^{(d)}}}$ whenever ${\bm{X}^{(d)}}$ is clear from the context.

An outlier score function $sc_{Def}$, or outlier score for simplicity, according to outlier definition $Def$, is a function $sc_{Def}:\mathbb{D}\times{\wp}(\mathbb{D})\mapsto\mathbb{S}$ that, given a set of $n$ objects, or dataset, $DS\subseteq\mathbb{D}$ (or $DS\in{\wp}(\mathbb{D})$, where ${\wp}(\mathbb{D})$ denotes the power set of $\mathbb{D}$), and an object $x\in{DS}$, returns a real number in the interval $\mathbb{S}\subseteq\mathbb{R}$, also said the outlier score (value) of $x$. The notation $sc_{Def}({x}^{(d)},{\bm{X}^{(d)}})$ is used to denote the outlier score of ${x}^{(d)}$ in a dataset whose elements are realizations of the random vector ${\bm{X}^{(d)}}$. The notation $sc_{Def}(x)$ is used when $DS$ or ${\bm{X}^{(d)}}$ are clear from the context.

Definition 3.2 (Outliers).

Given parameter $\alpha\in(0,1)$, the top-$\alpha$ outliers, or simply outliers whenever $\alpha$ is clear from the context, in a dataset $DS$ of $n$ points according to outlier definition $Def$, are the $\alpha n$ points of $DS$ associated with the largest values of score $sc_{Def}$.222Some definitions $Def$ associate outliers with the smallest values of score $sc_{Def}$. In these cases we assume to replace $Def$ with $Def^{\prime}$ having outlier score $sc_{Def^{\prime}}=-sc_{Def}$.

3.2. Kurtosis

The kurtosis is a measure of the tailedness of the probability distribution of a real-valued random variable, originating with Karl Pearson (Pearson, 1905; Fiori and Zenga, 2009; Westfall, 2014). Specifically, given random variable $X$, the kurtosis $\kappa_{X}$, or simply $\kappa$ whenever $X$ is clear from the context, is the fourth standardized moment of $X$

[TABLE]

Higher kurtosis is the result of infrequent extreme deviations or outliers, as opposed to frequent modestly sized deviations. Indeed, since kurtosis is the expected value of the standardized data raised to the fourth power. data within one standard deviation of the mean contribute practically nothing to kurtosis (note that raising a number that is less than $1$ to the fourth power makes it closer to zero), while the data values that almost totally contribute to kurtosis are those outside the above region, that is the outliers.

The lower bound is realized by the Bernoulli distribution with $0.5$ success probability, having kurtosis $\kappa=1$. Note that extreme platykurtic distributions, that is having kurtosis $\kappa=1$, have no outliers. However, there is no upper limit to the kurtosis of a general probability distribution, and it may be infinite.

The kurtosis of any univariate normal distribution is $3$, regardless of the values of its parameters. It is common to compare the kurtosis of a distribution to this value. Distributions with kurtosis equal to $3$ are called mesokurtic, or mesokurtotic. Distributions with kurtosis less than $3$ are said to be platykurtic. These distributions produce fewer and less extreme outliers than does the normal distribution. An example of a platykurtic distribution is the uniform distribution, which has kurtosis $1.8$. Distributions with kurtosis greater than $3$ are said to be leptokurtic. An example of a leptokurtic distribution is the Laplace distribution, which has tails that asymptotically approach zero more slowly than a normal distribution, and therefore produces more outliers than the normal distribution.

3.3. Theoretical cdf of the CFOF score

Next, we derive the theoretical cdf and pdf of the CFOF outlier score together with the expected score associated with a generic realization of an i.i.d. random vector.

Theorem 3.3.

Let ${DS}^{(d)}$ be a dataset consisting of realizations of an i.i.d. random vector ${\bm{X}^{(d)}}$ and let ${x}^{(d)}$ be an element of ${DS}^{(d)}$. For arbitrarily large dimensionalities $d$, the expected value and the cdf of the CFOF score are

[TABLE]

where $\kappa$ denotes the kurtosis of the random variable $X$.

Proof.

It is shown in (Angiulli, 2018) that, for arbitrary large values of $d$, the expected number of $k$-occurrences of ${x}^{(d)}$ in a dataset consisting of $n$ realizations of the random vector ${\bm{X}^{(d)}}$ is given by (see Theorem 30 of (Angiulli, 2018))

[TABLE]

Let $\tau(z_{x})$ denote the smallest integer $k^{\prime}$ such that ${\rm N}_{k^{\prime}}({x}^{(d)})\geq n\varrho$. By exploiting the equation above it can be concluded that

[TABLE]

Since $\mbox{CFOF}({x}^{(d)})=k^{\prime}/n=\tau(z_{x})/n$, the expression of Equation (4) follows by expressing moments $\mu_{2}$ and $\mu_{4}$ in terms of the kurtosis $\kappa={\mu_{4}}/{\mu_{2}^{2}}$.

For $\kappa=1$ the CFOF score is constant and does not depend on ${x}^{(d)}$ and $z_{x}$. However, for $\kappa>1$, since $\Phi(\cdot)$ is a cdf, the CFOF score is monotone increasing with the standard score $z_{x}$, thus $\mbox{CFOF}(x_{1}^{(d)})\leq\mbox{CFOF}(x_{2}^{(d)})$ if and only if $z_{x_{1}}\leq z_{x_{2}}$.

As for the cdf $Pr[\mbox{CFOF}({\bm{X}^{(d)}})\leq s]$ of the CFOF score, we need to determine for which values of ${x}^{(d)}$ the condition $\mbox{CFOF}({x}^{(d)})\leq s$ holds. By levaraging Equation (4)

[TABLE]

Consider the squared norm $\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}=\sum_{i=1}^{d}(X_{i}-\mu_{X})^{2}$. Since the squared norm is the sum of $d$ i.i.d. random variables, as $d\rightarrow\infty$, by the Central Limit Theorem, the distribution of the standard score of $\sum_{i=1}^{d}(X_{i}-\mu_{X})^{2}$ tends to a standard normal distribution. This implies that $\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}$ approaches a normal distribution with mean $\mu_{\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}}$ and standard deviation $\sigma_{\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}}$. Hence, for each $z\geq 0$,

[TABLE]

Thus, the probability $Pr[z_{x}\leq t(s)]$ converges to $\Phi(t(s))$, and the expression of Equation (5) follows. ∎

In the following, the cdf of the CFOF score will be denoted also as $F_{\rm CFOF}(s;\varrho,{\bm{X}^{(d)}})$, or simply $F_{\rm CFOF}(s)$ when $\varrho$ and the random vector ${\bm{X}^{(d)}}$ generating the dataset are clear from the context. As for the pdf of the CFOF score, $\forall\kappa>1$

[TABLE]

Figure 6 reports the theoretical cdf according to Equation (5) and the theoretical pdf according to Equation (8) of the CFOF score associated with an i.i.d. random vector uniformly distributed and normally distributed.

The following result provides a semantical characterization of CFOF outliers in large dimensional spaces.

Theorem 3.4.

Let ${DS}^{(d)}$ be a dataset consisting of realizations of an i.i.d. random vector ${\bm{X}^{(d)}}$. Then, for arbitrary large dimensionalities $d$, the CFOF outliers of ${DS}^{(d)}$ are the points associated with the largest squared norm standard scores.

Proof.

The result descends from the fact that the CFOF score is monotone increasing with the squared norm standard score, as shown in Theorem 3.3. ∎

3.4. Concentration of outlier scores

Definition 3.5 (Concentration of outlier scores).

Let $Def$ be an outlier definition with outlier score function $sc_{Def}$. We say that the outlier score $sc_{Def}$ concentrates if, for any i.i.d. random vector ${\bm{X}^{(d)}}$ having kurtosis $\kappa>1$, there exists a family of realizations ${x_{0}}^{(d)}$ of ${\bm{X}^{(d)}}$ ($d\in\mathbb{N}^{+}$) such that, for any $\epsilon\in(0,1)$ the following property holds333Note that if the above property holds “for any $\epsilon\in(0,1)$” it is also the case that it holds “for any $\epsilon>0$”. We preferred to use the former condition for symmetry with respect to the scores smaller than $sc_{Def}({x}^{(d)})$. Indeed, for any $\epsilon\geq 1$ the absolute value function can be removed from Equation (9).

[TABLE]

that is to say the probability to observe a score value $sc_{Def}({\bm{X}^{(d)}})$ having relative distance not greater than $\epsilon$ from the reference score $sc_{Def}({x_{0}}^{(d)})$ tends to $1$ as the dimensionality $d$ goes to infinity.

Theorem 3.6.

Let $Def$ be an outlier definition with outlier score function $sc_{Def}$ which is monotone increasing with respect to the squared norm standard score. Then, the outlier score $sc_{Def}$ concentrates if and only if, for any i.i.d. random vector ${\bm{X}^{(d)}}$ having kurtosis $\kappa>1$ and for any $\epsilon\in(0,1)$, the family of realizations ${x_{0}}^{(d)}$ of ${\bm{X}^{(d)}}$ having the property that $z_{x_{0}}=0$, is such that

[TABLE]

where ${x_{1}}^{(d)}$ and ${x_{2}}^{(d)}$ satisfy the following conditions

[TABLE]

Proof.

As already pointed out in the proof of Theorem 3.3, as $d\rightarrow\infty$, the distribution of the standard score $Z=(Y-\mu_{Y})/\sigma_{Y}$ of the squared norm $Y=\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}$ tends to a standard normal distribution and, hence, for each $z\geq 0$, $Pr[Z\leq z]=\Phi(z)$. Since the $sc_{Def}$ score is by assumption monotone increasing with the squared norm standard score, for arbitrarily large values of $d$ and for any realization ${x}^{(d)}$ of ${\bm{X}^{(d)}}$,

[TABLE]

If $sc_{Def}$ concentrates, then outlier score values must converge towards the outlier score of the points ${x_{0}}^{(d)}$ whose squared norm standard score $z_{x_{0}}$ corresponds to the expected value $\mathbf{E}[Z]=0$ of $Z$. Given $\epsilon>0$, in order to hold

[TABLE]

it must be the case that

[TABLE]

where ${x_{1}}^{(d)}$ and ${x_{2}}^{(d)}$ are defined as in the statement of the theorem. The result then follows from the last condition. ∎

Now we show that the separation between the CFOF scores associated with outliers and the rest of the CFOF scores is guaranteed in any arbitrary large dimensionality.

Theorem 3.7.

For any fixed $\varrho\in(0,1)$, the CFOF outlier score does not concentrate.

Proof.

Let ${\bm{X}^{(d)}}$ be an i.i.d. random vector having kurtosis $\kappa>1$. As far as the CFOF score is concerned, from Equation (4), the point ${x_{1}}^{(d)}$ defined in Theorem 3.6 is such that

[TABLE]

Note that for any fixed $\varrho\in(0,1)$, $\Phi^{-1}(\varrho)$ is finite. Moreover, since $z_{x_{0}}=0$, from Equation (4) it holds that $\mbox{CFOF}({x_{0}}^{(d)})\in(0,0.5)$. Hence, $\Phi(z_{x_{1}})\rightarrow 0$ if and only if $z_{x_{1}}\rightarrow-\infty$ if and only if $\Phi^{-1}((1-\epsilon)\mbox{CFOF}(x_{0}^{(d)}))\rightarrow-\infty$ if and only if $(1-\epsilon)\mbox{CFOF}(x_{0}^{(d)})\rightarrow 0$ if and only if $\epsilon\rightarrow 1$.

Thus, for arbitrary large dimensionalities $d$, for each realization ${x_{0}}^{(d)}$ of ${\bm{X}^{(d)}}$ there exists $\epsilon>0$ such that $Pr[\left|(\mbox{CFOF}({\bm{X}^{(d)}})-\mbox{CFOF}({x_{0}}^{(d)}))/\mbox{CFOF}({x_{0}}^{(d)})\right|<\epsilon]<1$ and, hence, the CFOF outlier score does not concentrate. ∎

Note that, with parameter $\varrho\in(0,1)$, the CFOF score does not concentrate for both bounded and unbounded sample sizes.

Figure 7 compares the empirical CFOF scores associated with the squared norm standard score of a normal dataset for increasing values of $d$ and the theoretical CFOF associated with arbitrarily large dimensionalities ($d=\infty$). It can be seen that for large $d$ values the scores tend to comply with the value predicted by Equation (4). Moreover, as predicted by Theorem 3.7, the concentration of the CFOF scores is avoided in any dimensionality.

3.5. The effect of the data kurtosis

As recalled above, the kurtosis $\kappa$ is such that $\kappa\in[1,+\infty]$. For extreme platykurtic distributions (i.e., having $\kappa\rightarrow 1^{+}$) the CFOF score tends to $\varrho$, hence to a constant, while for extreme leptokurtic distributions (i.e., having $\kappa\rightarrow\infty$) the CFOF score tends to the cumulative distribution function of the standard normal distribution.

Theorem 3.8.

Let ${x}^{(d)}$ be a realization of an i.i.d. random vector ${\bm{X}^{(d)}}$ having, w.l.o.g., null mean. Then, for arbitrary large dimensionalities $d$,

[TABLE]

Proof.

The two expression can be obtained by exploiting the closed form of the cdf of the CFOF score reported in Equation (4). ∎

Note that extreme platykurtic distributions have no outliers and, hence, are excluded by the definition of concentration of outlier score. For $\kappa=1$, CFOF scores are constant, and this is consistent with the absolute absence of outliers.

Figure 8 reports the CFOF scores associated with different kurtosis values $\kappa\in\{1.8,~{}3,~{}9,~{}100,~{}\infty\}$. Curves are obtained by leveraging Equation (5) and setting $\varrho=0.01$. The curves on the left represent the cumulative distribution function of the CFOF scores. For infinite kurtosis the CFOF scores are uniformly distributed between [math] and $1$, since, from Equation (5), for $\kappa\rightarrow\infty$, $Pr[\mbox{CFOF}({\bm{X}^{(d)}})\leq s]=s$. Moreover, the curves highlight that the larger the kurtosis of the data and the larger the probability to observe higher scores. The curves on the right represent the score values ranked in decreasing order. The abscissa reports the fraction of data points in logarithmic scale. These curves allow to visualize the fraction of data points whose score will be above a certain threshold.

3.6. Behavior in presence of different distributions

Despite the apparent similarity of the CFOF score with distance-based scores, these two families of scores are deeply different and, while distance-based outliers can be categorized among to the global outlier scores, CFOF shows adaptivity to different density levels, a characteristics that makes it more similar to local outlier scores.

This characteristics depends in part on the fact that actual distance values are not employed in the computation of the score. Indeed, the CFOF score is invariant to all of the transformations that do not change the nearest neighbor ranking, such as translating the data or scaling the data.

To illustrate, consider Figure 9 showing a dataset consisting of two normally distributed clusters, each consisting of $250$ points. The cluster centered in $(4,4)$ is obtained by translating and scaling by a factor $0.5$ the cluster centered in the origin. The top $25$ CFOF outliers for $k_{\varrho}=20$ are highlighted. It can be seen that the outliers are the “same” objects of the two clusters. Notice that a similar behavior can be observed, that is outliers will emerge both in the sparser regions of the space and along the borders of clusters, also when the clusters are not equally populated, provided that they contain at least $k_{\varrho}$ objects.

Now we will provide theoretical evidence that the above discussed properties of the CFOF score are valid in any arbitrary large dimensionality.

Definition 3.9 (Translation-invariant and homogeneous outlier score).

Let ${\bm{X}^{(d)}}$ an i.i.d. random vector, let $a\in\mathbb{R}$ and ${b}^{(d)}\in\mathbb{R}^{d}$, and let ${\bm{Y}^{(d)}}=a{\bm{X}^{(d)}}+{b}^{(d)}$. An outlier score $sc_{Def}$ is said to be translation-invariant and homogeneous if, for all the realizations ${x}^{(d)}$ of ${\bm{X}^{(d)}}$, it holds that

[TABLE]

Theorem 3.10.

For arbitrary large dimensionalities $d$, the CFOF score is translation-invariant and homogeneous.

Proof.

We note that the squared norm standard score $z_{x,{\bm{X}^{(d)}}}$ of ${x}^{(d)}$ is identical to the squared norm standard score $z_{y,{\bm{Y}^{(d)}}}$ of ${y}^{(d)}=a{x}^{(d)}+{b}^{(d)}$. Indeed, the mean of ${\bm{Y}^{(d)}}$ is the vector ${\mu}^{(d)}_{Y}=(\mu_{Y_{1}},\mu_{Y_{2}},\ldots,\mu_{Y_{d}})$, where $\mu_{Y_{i}}=\mathbf{E}[aX_{i}+b_{i}]=a\mu_{X}+b_{i}$ ($1\leq i\leq d$). As for $\|{y}^{(d)}-{\mu}^{(d)}_{Y}\|^{2}=\|(a{x}^{(d)}+{b}^{(d)})-(a\mu_{X}+{b}^{(d)})\|^{2}=\|a({x}^{(d)}-\mu_{X})\|^{2}=a^{2}\|{x}^{(d)}-\mu_{X}\|^{2}$. Analogously, $\|{\bm{Y}^{(d)}}-\mu_{Y}\|^{2}=a^{2}\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}$, and $\mu_{\|{\bm{Y}^{(d)}}-\mu_{Y}\|^{2}}=a^{2}\mu_{\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}}$ and $\sigma_{\|{\bm{Y}^{(d)}}-\mu_{Y}\|^{2}}=a^{2}\sigma_{\|{\bm{X}^{(d)}}-\mu_{X}\|^{2}}$. Hence,

[TABLE]

Moreover, we recall that the $k$th central moment has the following two properties: $\mu_{k}(X+b)=\mu_{k}(X)$ (called translation-invariance), and $\mu_{k}(aX)=a^{k}\mu_{k}(X)$ (called homogeneity). Hence, $\mu_{k}(Y)=\mu_{k}(Y_{i})=\mu_{k}(aX+b_{i})=a^{k}\mu_{k}(X)$. Since variables $Y_{i}$ have identical central moments, by applying the above property to Equation (5), the statement eventually follows:

[TABLE]

∎

Next, we introduce the concept of i.i.d. mixture random vector as a tool for modeling an intrinsically high-dimensional dataset containing data populations having different characteristics.

An i.i.d. mixture random vector ${\bm{Y}^{(d)}}$ is a random vector defined in terms of $K$ i.i.d. random vectors ${\bm{Y_{1}}^{(d)}},{\bm{Y_{2}}^{(d)}},\ldots,{\bm{Y_{K}}^{(d)}}$ with associated selection probabilities $\pi_{1},\pi_{2},\ldots,\pi_{K}$, respectively. Specifically, for each $i$, with probability $\pi_{i}$ the random vector ${\bm{Y}^{(d)}}$ assumes value ${\bm{Y_{i}}^{(d)}}$.

A dataset $DS$ generated by an i.i.d. mixture random vector ${\bm{Y}^{(d)}}$, consists of $n$ points partitioned into $K$ clusters $C_{1},C_{2},\ldots,C_{K}$ composed of $n_{1},n_{2},\ldots,n_{K}$ points, respectively, where each $C_{i}$ is formed by realizations of the random vector ${\bm{Y_{i}}^{(d)}}$ selected with probability $\pi_{i}$ ($1\leq i\leq K$).

Given a set of clusters $C_{1},C_{2},\ldots,C_{K}$, we say that they are non-overlapping if, for each cluster $C_{i}$ and point $x\in C_{i}$, ${\rm NN}_{n_{i}}(x,DS)=C_{i}$.

The following result clarifies how CFOF behaves in presence of clusters having different densities.

Theorem 3.11.

Let $DS$ be a dataset of $K$ non-overlapping clusters $C_{1},C_{2},\ldots,C_{K}$ generated by the i.i.d. mixture random vector ${\bm{Y}^{(d)}}$, let $\varrho\leq\min_{i}\{\pi_{i}\}$, let $O$ be the top-$\alpha$ CFOF outliers of $DS$, and let $O=O_{1}\cup O_{2}\cup\ldots\cup O_{K}$ be the partition of $O$ induced by the clusters of $DS$. Then, for arbitrary large dimensionalities $d$, each $O_{i}$ consists of the top-$(\alpha_{i}/\pi_{i})$ $(\varrho/\pi_{i})$–CFOF outliers of the dataset $C_{i}$ generated by the i.i.d. random vector ${\bm{Y_{i}}^{(d)}}$ ($1\leq i\leq K$), where

[TABLE]

and $s^{\ast}$ is such that $F_{\rm CFOF}(s^{\ast};\varrho,{\bm{Y}^{(d)}})=1-\alpha$.

Proof.

Since the clusters are non-overlapping, the first $n_{i}$ direct and reverse neighbors of each point ${x}^{(d)}\in C_{i}$ are points belonging to the same cluster $C_{i}$ of ${x}^{(d)}$. Hence, being $\varrho n\leq n_{i}$, the CFOF score of point ${x}^{(d)}$ depends only on the points within its cluster. Thus the score $s=k/n$ with respect to the whole dataset can be transformed into the score $s_{i}=k/n_{i}=(k/n)(n/n_{i})=s/\pi_{i}$ with respect the cluster $C_{i}$. Analogously, the parameter $\varrho$ with respect to the whole dataset can be transformed into the parameter $\varrho_{i}$ with respect to the cluster $C_{i}$, by requiring that $\varrho n=\varrho_{i}n_{i}$, that is $\varrho_{i}=\varrho(n/n_{i})=\varrho/\pi_{i}$. Thus, the cdf of the CFOF score can be formulated as

[TABLE]

Consider now the score value $s^{\ast}$ such that $F_{\rm CFOF}(s^{\ast};\varrho,{\bm{Y}^{(d)}})=1-\alpha$. Then the expected number of outliers from cluster $C_{i}$ is $\alpha_{i}n=n\pi_{i}\left(1-F_{\rm CFOF}(s^{\ast}/\pi_{i};\varrho/\pi_{i},{\bm{Y_{i}}^{(d)}})\right)$, from which the result follows. ∎

Thus, from the above result it can be concluded that the number of outliers coming from each cluster $C_{i}$ is related both to its generating distribution ${\bm{Y_{i}}^{(d)}}$ and to its relative size $\pi_{i}$.

As for the generating distribution, if we consider points having positive squared norm standard score (which form the most extreme half of the population), then at the same squared norm standard score value, the CFOF score is higher for points whose generating distribution has larger kurtosis.

Theorem 3.12.

Let ${\bm{X}^{(d)}}$ and ${\bm{Y}^{(d)}}$ be two i.i.d. random vectors, and let ${x}^{(d)}$ and ${y}^{(d)}$ be two realizations of ${\bm{X}^{(d)}}$ and ${\bm{Y}^{(d)}}$, respectively, such that $z_{x,{\bm{X}^{(d)}}}=z_{y,{\bm{Y}^{(d)}}}\geq 0$. Then, for arbitrary large dimensionalities $d$, $\mbox{CFOF}({x}^{(d)},{\bm{X}^{(d)}})\geq\mbox{CFOF}({y}^{(d)},{\bm{Y}^{(d)}})$ if and only if $\kappa_{X}\geq\kappa_{Y}$.

Proof.

The result descends from the fact that for non-negative $z\geq 0$ values, the CFOF score is monotone increasing with respect to the kurtosis parameter. ∎

Thus, the larger the cluster kurtosis, the larger the fraction of outliers coming from that cluster. To clarify the relationship between kurtosis and number of outliers we considered a dataset consisting of two clusters: the first (second, resp.) cluster consists of points generated according to an i.i.d. random vector having kurtosis $\kappa_{1}=3$ ($\kappa_{2}\in\{1.8,3,9\}$, resp.). We then varied the occurrence probability $\pi_{1}$ of the first cluster in the interval $[0.2,0.8]$, while the occurrence probability of the second cluster is $\pi_{2}=1-\pi_{1}$, and set $\varrho$ to $0.01$ and the fraction $\alpha$ of outliers to be selected to $0.05$. Figure 10 reports the percentage of the top-$\alpha$ outliers that come from the first cluster (that is $\alpha_{1}^{\prime}=\frac{\alpha_{1}}{\alpha}\cdot 100$) as a function of $\pi_{1}$ for different $\kappa_{2}$ values. It can be seen that, for $\kappa_{1}=\kappa_{2}$ the fraction $\alpha_{1}^{\prime}$ is more closely related to $\pi_{1}$ (with $\alpha_{1}^{\prime}=0.5$ for $\pi_{1}=0.5$), while in the general $\alpha_{1}^{\prime}$ is directly proportional to the ratio $\frac{\kappa_{1}}{\kappa_{2}}$.

Intuitively, for datasets consisting of multiple shifted and scaled copies of a given cluster, a score able to retrieve local outliers should return the same outliers from each cluster, both in terms of their number and of their location within the cluster. Next we show that this is indeed the case of the CFOF score in any arbitrary large dimensionality.

We say that a collection of sets $S_{1},S_{2},\ldots,S_{K}$ is balanced, if $|S_{1}|=|S_{2}|=\ldots=|S_{K}|$.

We say that an i.i.d. mixture random vector ${\bm{Y}^{(d)}}$ is homogeneous if there exist random vector ${\bm{Y_{0}}^{(d)}}$, real numbers $a_{i}\in\mathbb{R}$, and vectors ${b_{i}}^{(d)}\in\mathbb{R}^{d}$, such that ${\bm{Y_{i}}^{(d)}}=a_{i}{\bm{Y_{0}}^{(d)}}+{b_{i}}^{(d)}$ ($1\leq i\leq K$).

Theorem 3.13.

Let ${DS}^{(d)}$ be a dataset composed of balanced non-overlapping clusters generated by an homogeneous i.i.d. mixture random vector ${\bm{Y}^{(d)}}$. Then, for arbitrary large dimensionalities $d$, the partition of the CFOF outliers of ${DS}^{(d)}$ induced by the clusters of ${DS}^{(d)}$ is balanced.

Proof.

Since clusters are balanced, $\pi_{i}=1/K$. Moreover, by Theorems 3.10 and 3.11,

[TABLE]

which means that, for any score value $s$, the number of expected outliers from any cluster is the same. ∎

We call semi–local any outlier definition for which the property stated in the statement of Theorem 3.13 holds for homogeneous i.i.d. mixture random vectors, but not for non-homogeneous i.i.d. mixture random vectors.

3.7. On the concentration properties of distance-based and density-based

outlier scores

We have already discussed the distance concentration phenomenon, that is the tendency of distances to become almost indiscernible as dimensionality increases, and empirically shown that the concentration effect also affects different families of outlier scores (see Section 2.2). In this section we theoretically assess the concentration properties of some of these scores, by formally proving that distance-based and density outlier scores concentrate according to Definition 3.5. Specifically, we consider the KNN (Ramaswamy et al., 2000), aKNN (Angiulli and Pizzuti, 2002), and LOF (Breunig et al., 2000) outlier scores.

Intuitively, the above results descend by the specific role played by distances in these definitions. Indeed, distance-based outlier scores detect outliers on the basis of the distance separating points from their nearest neighbors, a measure whose discrimination capability, due to the distance concetration effect, is expected to become more and more feeble as the dimensionality increases. As for density-based outliers, in order to capture a notion of local (or normalized) density, they compute the ratio between the distance separating a point from its nearest neighbors and the average of the same measure associated with their nearest neighbors. Being the two above terms subject to concentration of distances, their ratio is expected to become closer and closer to $1$ as the dimensionality increases.

Formal details, including the role played by parameters, are provided in subsequent Theorems 3.14, 3.15, and 3.16.

Theorem 3.14.

For any fixed integer number $k$ such that $0<k<n$ (for any fixed real number $\varrho\in(0,1)$, resp.), the KNN outlier score with parameter $k$ ($k=\varrho n$, resp.) concentrates.

Proof.

Let ${\bm{X}^{(d)}}$ be an i.i.d. random vector having kurtosis $\kappa>1$. W.l.o.g., assume that $\mu_{X}=0$, then from (Angiulli, 2018) (see Lemma 23)

[TABLE]

For any fixed $k$ such that $0<k<n$ ($\varrho\in(0,1)$, resp.), it is the case that $0<k/n<1$ and $\Phi^{-1}(k/n)$ is finite. Moreover, for arbitrary large dimensionalities $d$, $\sqrt{O(\|{x}^{(d)}\|^{2},d)+\sqrt{O(\|{x}^{(d)}\|^{2},d)}}\approx\sqrt{O(\|{x}^{(d)}\|^{2},d)}$ and, then

[TABLE]

Since $\|{x}^{(d)}\|^{2}=z_{x}\sigma_{\|{X}^{(d)}\|^{2}}+\mu_{\|{X}^{(d)}\|^{2}}=z_{x}\sqrt{d(\mu_{4}-\mu_{2}^{2})}+d\mu_{2}$, by substituting

[TABLE]

From the above expressions, the KNN score is monotone increasing with the squared norm standard score. Then, Theorem 3.6 can be applied. Consider the family ${x_{0}}^{(d)}$ ($d\in\mathbb{N}^{+}$) of realizations of ${\bm{X}^{(d)}}$ such that $\|{x_{0}}^{(d)}\|^{2}=\mu_{\|{X}^{(d)}\|^{2}}=d\mu_{2}$. Then, the point ${x_{1}}^{(d)}$ defined in Theorem 3.6 is such that

[TABLE]

As for the point ${x_{2}}^{(d)}$ defined in Theorem 3.6, for symmetry it is such that

[TABLE]

Now, consider any $\epsilon\in(0,1)$. Then, $z_{x_{1}}=O(-\sqrt{d})$ is negative and $z_{x_{2}}=O(\sqrt{d})$ is positive. Moreover, for $d\rightarrow\infty$, $z_{x_{1}}$ diverges to $-\infty$ and $z_{x_{2}}$ diverges to $+\infty$ and, hence

[TABLE]

and the statement follows. ∎

As for the effect of the kurtosis on the concentration of distance-based and density-based scores scores, from the expressions of $z_{x_{1}}$ and $z_{x_{2}}$ in Theorem 3.14, we can conclude the convergence rate towards concentration of these scores is inversely proportional to the square root of the data kurtosis.

Theorem 3.15.

For any fixed integer number $k$ such that $0<k<n$ (for any fixed real number $\varrho\in(0,1)$, resp.), the aKNN outlier score with parameter $k$ ($k=\varrho n$, resp.) concentrates.

Proof.

Recall that

[TABLE]

and, hence, aKNN can be regarded as the sum of $k$ KNN scores, each one associated with a different parameter ranging in $\{1,2,\ldots,k\}$. Since, by Theorem 3.14, the KNN score concentrates, it must be the case that also the sum of $k$ of these KNN scores must concentrate and, hence, also the aKNN score concentrates. ∎

Theorem 3.16.

For any fixed integer number $k$ such that $0<k<n$ (for any fixed real number $\varrho\in(0,1)$, resp.), the LOF outlier score with parameter $k$ ($k=\varrho n$, resp.) concentrates.

Proof.

The LOF score is defined as

[TABLE]

where

[TABLE]

are the $k$-distance (k–dist), the reachability-distance (r–dist), and the local reachability distance (lr–dist), respectively. This score assigns value $1$ to points inside clusters and score significantly larger than $1$ to outliers.

Consider outlier points ${x}^{(d)}$. Since outliers are not neighbors of their nearest neighbors, that is ${\rm dist}({x}^{(d)},{\it nn}_{i}({x}^{(d)}))>\textit{k--dist}({\it nn}_{i}({x}^{(d)}))$ for $i\leq k$, then the numerator of LOF must coincide with $\mbox{aKNN}({x}^{(d)})$. As for the denominator, note that for any point ${x}^{(d)}$, $\mbox{aKNN}({\it nn}_{i}({x}^{(d)}))\leq\textit{lr--dist}({\it nn}_{i}({x}^{(d)}))$. Hence, for ${x}^{(d)}$ an outlier point

[TABLE]

The concentration of LOF then follows from the concentration of the aKNN score. ∎

Summarizing, from the above analysis it follows that distance-based and density-based scores concentrate in the following cases:

• —

sample size $n$: both bounded ($n>k$ finite) and unbounded ($n\rightarrow\infty$);

• —

parameter $k$: both fixed ($k={\rm const.}$) and variable ($k=\varrho n$ with $\varrho\in(0,1)$).

Moreover, note that for finite $n$, these score concentrate even for $k=n$ (or, equivalently, $\varrho=1)$.444 Consider the proof of Theorem 3.14. Even if for $k=n$ the term $\Phi^{-1}(k/n)$ evaluates to $+\infty$, this can be considered valid only in the case of infinite sample sizes. For finite samples, the contribution to the KNN score associated with the standard deviation $\sigma_{\|{x}^{(d)}-{\bm{X}^{(d)}}\|^{2}}$ of the squared distance must be finite and, hence, can be ignored, as already done in the proof of Theorem 3.14 for $k<n$. Thus, the only way to avoid concentration of distance-based and density-based outlier scores is to consider infinite samples, that is $n=\infty$, and to employ the parameter $k=\infty$, or equivalently $\varrho=1$.555In this case distance-outliers coincide with the points located on the boundaries of the data distribution support. If the support is infinite, outliers are located at infinity.

3.8. On the concentration properties of

reverse nearest neighbor-based outliers

Consider the following result from (Newman et al., 1983; Angiulli, 2018): Let $k>0$ be a fixed natural number, then

[TABLE]

where the convergence is in distribution.

Intuitively, the above result states that if $k$ is held fixed, while both the dimensionality $d$ and the sample size $n$ tend to infinity, then the $k$-occurrences function will tend to be identically equal to zero. This is a consequence of the hubness phenomenon, since only a few points, the hubs, located in proximity of the mean, will be selected as $k$ nearest neighbors by any other point. Thus, the RNNc score for fixed parameter $k$ and unbounded sample sizes $n$ is subject to concentration.666This also means that we can enforce CFOF to concentrate only by allowing inconsequentially small reverse neighborhood sizes $\varrho\rightarrow 0$ in presence of unboundedly large samples $n\rightarrow\infty$.

Moreover, by Equation (6), the concentration of the RNNc scores can be avoided by relating the parameter $k$ to the sample size. This behavior of RNNc scores has been already observed in the literature (Radovanović et al., 2015), by noticing that for RNNc the discrimination of scores represents a notable weakness of the methods, with two contributing factors: hubness and inherent discreteness. Thus, in order to add more discrimination to RNNc they suggested to raise $k$, possibly to some value comparable with $n$, but with two concerns: ($i$) with increasing $k$ the notion of outlier moves from local to global, thus if local outliers are of interest they can be missed; ($ii$) $k$ values comparable with $n$ raise issues with computational complexity.

It is important to highlight that the behavior of the CFOF score is deeply different from that of the RNNc scores, since CFOF is not affected at all by the two above mentioned problems. Indeed, CFOF outliers are well separated from inliers also for relatively small values of the parameter $\varrho$.

To illustrate, Figure 11 reports for arbitrary large dimensionalities, the CFOF (red increasing curves) and RNNc scores (blue decreasing curves) as a function of the squared norm standard score $z$. As for the green dashed (normally distributed) curve, it is proportional to the fraction of the data population having the squared norm standard score reported on the abscissa.

Figure 11a is concerns i.i.d. random vectors having distribution with kurtosis $\kappa=3$ (as the normal distribution) for different values of $\varrho=k/n$, namely $\varrho=0.0001$ (dash-dotted curves), $\varrho=0.001$ (solid curves), and $\varrho=0.01$ (dashed curves). Since outliers are the points associated with largest values of $z$, it is clear that while CFOF is able to establish a clear separation between outliers and inliers for any value of the parameter $\varrho$, the ${\rm N}_{k}$ function is close to zero for $z\geq 0$, thus presenting a large false positive rate.

Figure 11b reports the CFOF and RNNc scores associated with i.i.d. random vectors having cdfs with different kurtosis values, namely $\kappa=1.8$ (as the uniform distribution), $\kappa=3$ (as the normal distribution), and $\kappa=9$ (as the exponential distribution) when $\varrho=k/n=0.001$.

Figures 11c and 11d report the scores in logarithmic scale, to appreciate differences between ${\rm N}_{k}$ values. According to Equation (6), for $z\geq 0$ the RNNc score is monotonically decreasing with the kurtosis. However, the figures point out that in this case the RNNc outlier scores are infinitesimally small and even decreasing of orders of magnitude with the kurtosis. By applying the same line of reasoning of Theorem 3.10 and Theorem 3.13 to Equation (6), we can show that RNNc is both translation-invariant and homogeneous and semi–local. Unfortunately, it must be also pointed out that the hubness phenomenon affecting the RNNc score and the discussed behavior of RNNc for different kurtosis values, make the above properties immaterial for many combinations of $n$ and $k$, and for many combinations of clusters kurtoses.

Consider the standard deviation $\sigma^{out}_{z_{0}}(sc)$ of the outlier score $sc$ distribution starting from $z_{0}$:

[TABLE]

where, $\mu^{out}_{z_{0}}(sc)$ represents the mean of the above distribution. Intuitively, $\sigma^{out}_{z_{0}}(sc)$ measures the amount of variability of the score values $sc$ associated with the most extreme observations of the data population. Thus, the ratio

[TABLE]

called separation for short in the following, measures how larger is the variability of the scores of the CFOF outliers with respect to that of ODIN.

Figure 12 shows the separation for arbitrary large dimensionalities, for kurtosis $\kappa\in[1,9]$, and for $z_{0}=0$ (Figure 12a) and $z_{0}=3$ (Figure 12b). According to the values reported in the plots, the separation is of several orders of magnitude and, moreover, the smaller the parameter $\varrho$, the larger its value.

4. The fast-CFOF algorithm

In general, CFOF scores can be determined in time $O(n^{2}d)$, where $d$ denotes is the dimensionality of the feature space (or the cost of computing a distance), after computing all pairwise dataset distances.

In this section, we introduce the fast-CFOF technique for dealing with very large and high-dimensional datasets. As its main peculiarity, fast-CFOF does not require the computation of the exact nearest neighbor sets and, from the computational point of view, does not suffer of the curse of dimensionality affecting nearest neighbor search techniques.

The technique builds on the following probabilistic formulation of the CFOF score. Let ${\bm{X}^{(d)}}$ be a random vector from an unknown probability law $p(\cdot)$. Given parameter $\varrho\in(0,1)$, the (Probabilistic) Concentration Outlier Factor CFOF of the realization ${x}^{(d)}$ of ${\bm{X}^{(d)}}$ is defined as follows:

[TABLE]

where the probability has to be considered conditional on ${x}^{(d)}$ being drawn in a sample of $n$ realizations of the random vector ${\bm{X}^{(d)}}$. The above probabiliy is estimated by using the empirical distribution of the data as $p(\cdot)$. To differentiate the two definitions reported in Equations (1) and (10), we also refer to the former as hard-CFOF and to the latter as soft-CFOF.

Intuitively, the soft-CFOF score measures how many neighbors have to be taken into account in order for the expected number of dataset objects having it among their neighbors to correspond to a fraction $\varrho$ of the overall population.

The rationale of Equation (10) is that of providing a definition tailored to leverage sampling for determining CFOF outliers. Indeed, as discussed in the following section describing the fast-CFOF algorithm, the probability $Pr[{x}^{(d)}\in{\rm NN}_{k^{\prime}}({\bm{X}^{(d)}})]$ is estimated by looking at the number of observations having ${x}^{(d)}$ among their $k^{\prime}$ nearest neighors in a sample of size $s$ of the whole dataset. However, we point out that if $s$ is equal to the sample size $n$, then Equation (10) reduces to Equation (1), that is soft-CFOF and hard-CFOF coincide. Indeed, in this case the above probability is smaller $\varrho$ for $k^{\prime}<n\cdot\mbox{CFOF}({x}^{(d)})$ and not smaller than $\varrho$ for $k^{\prime}\geq n\cdot\mbox{CFOF}({x}^{(d)})$. This also means that the (exact) CFOF algorithm coincides with fast-CFOF for $s=n$.

4.1. The fast-CFOF technique

The fast-CFOF algorithm aims at improving efficiency by estimating soft-CFOF scores.

Given a dataset ${\bf DS}$ and two objects $x$ and $y$ from ${\bf DS}$, the building block of the algorithm is the computation of the probability $Pr[x\in{\rm NN}_{k}(y)]$. Consider the boolean function $B_{x,y}(z)$ defined on instances $z$ of ${\bf DS}$ such that $B_{x,y}(z)=1$ if $z$ lies within the hyper-ball of center $y$ and radius ${\rm dist}(x,y)$, and [math] otherwise. We want to estimate the average value $\overline{B}_{x,y}$ of $B_{x,y}$ in ${\bf DS}$, that is the ratio of instances $z\in{\bf DS}$ such that $B_{x,y}(z)=1$ holds, which corresponds to the probability $p(x,y)$ that a randomly picked dataset object $z$ is at distance not grater than ${\rm dist}(x,y)$ from $y$, that is ${\rm dist}(y,z)\leq{\rm dist}(x,y)$.

In particular, for our purposes it is enough to compute $\overline{B}_{x,y}$ within a certain error bound. Thus, we resort to the batch sampling technique, which consists in picking up $s$ elements of ${\bf DS}$ randomly and estimating $p(x,y)=\overline{B}_{x,y}$ as the fraction of the elements of the sample satisfying $B_{x,y}$ (Watanabe, 2005).

Let $\hat{p}(x,y)$ the value of ${p}(y,x)$ estimated by means of the sampling procedure. Given $\delta>0$, an error probability, and $\epsilon$, $0<\epsilon<1$, an absolute error, if the size $s$ of the sample satisfies certain conditions (Watanabe, 2005) the following relationship holds

[TABLE]

For large values of $n$, it holds that the variance of the Binomial distribution becomes negligible with respect to the mean and the cumulative distribution function $\textit{binocdf}(k;p,n)$ tends to the step function H\big{(}k-np\big{)}, where H\big{(}k\big{)}=0 for $k<0$ and H\big{(}k\big{)}=1 for $k>0$. Thus, we can approximate the value $Pr[x\in{\rm NN}_{k}(y)]=\textit{binocdf}(k;p(x,y),n)$ with the boolean function H\big{(}k-k_{up}(x,y)\big{)}, with $k_{up}(x,y)=n\widehat{p}(x,y)$.777Alternatively, by exploiting the Normal approximation of the Binomial distribution, a suitable value for $k_{up}(x,y)$ is given by $k_{up}(x,y)=n\widehat{p}(x,y)+c\sqrt{n\widehat{p}(x,y)(1-\widehat{p}(x,y))}$ with $c\in[0,3]$. It then follows that we can obtain $Pr[x\in{\rm NN}_{k}(\textbf{X})]$ as the average value of the boolean function H\big{(}k-n\widehat{p}(x,y)\big{)}, whose estimate can be again obtained by exploiting batch sampling. Specifically, fast-CFOF exploits one single sample in order to perform the two estimates above described.

The pseudo-code of fast-CFOF is reported in Algorithm 1. The algorithm receives in input a list $\bm{\varrho}=\varrho_{1},\ldots,\varrho_{\ell}$ of values for the parameter $\varrho$, since it is able to perform a multi-resolution analysis, that is to simultaneously compute dataset scores associated with different values of the parameter $\varrho$, with the same asymptotic temporal cost. We assume $\bm{\varrho}=0.001,0.005,0.01,0.05,0.1$ and $\epsilon,\delta=0.01$ as default values for the parameters. See later for details on the effect of the parameters.

First, the algorithm determines the size $s$ of the sample (or partition) of the dataset needed in order to guarantee the bound reported in Eq. (11). We notice that the algorithm does not require the dataset to be entirely loaded in main memory, since only one partition at a time is needed to carry out the computation. Thus, the technique is suitable also for disk resident datasets. Moreover, we assume that the position of the objects within the dataset has no connection with their spatial relationship, for otherwise a preprocessing step consisting in randomizing the dataset is required (Sanders, 1998).

Then, each partition consisting of a group of $s$ consecutive objects, is processed by fast-CFOF_part, whose pseudo-code is reported in Procedure 2. This subroutine estimates CFOF scores of the objects within the partition through batch sampling.

The matrix $hst$, consisting of $s\times B$ counters, is employed by fast-CFOF_part. The entry $hst(i,k)$ of $hst$ is used to estimate how many times the sample object $x^{\prime}_{i}$ is the $k$th nearest neighbor of a generic object dataset. Values of $k$, ranging from $1$ to $n$, are partitioned into $B$ log-spaced bins. The function k_bin maps original $k$ values to the corresponding bin, while $\textit{k\_bin}^{-1}$ implements the reverse mapping by returning a certain value within the corresponding bin.

For each sample object $x^{\prime}_{i}$ the distance $dst(j)$ from any other sample object $x^{\prime}_{j}$ is computed (lines 3-4) and, then, distances are ordered (line 5) obtaining the list $ord$ of sample identifiers such that $dst(ord(1))\leq dst(ord(2))\leq\ldots\leq dst(ord(s))$. Notice that, since we are interested only in ranking distances, as far as distance computations is concerned, in the Euclidean space the squared distance can be employed to save time.

Moreover, for each element $ord(j)$ of $ord$, the variable $p$ is set to $j/s$ (line 7), representing the probability $p(x^{\prime}_{ord(j)},x^{\prime}_{i})$, estimated through the sample, that a randomly picked dataset object is located within the region of radius $dst(ord(j))=dist(x^{\prime}_{i},x^{\prime}_{ord(j)})$ centered in $x^{\prime}_{i}$. The value $k_{up}$ (line 8) represents the point of transition from [math] to $1$ of the step function H\big{(}k-k_{up}\big{)} employed to approximate the probability $Pr[x^{\prime}_{ord(j)}\in{\rm NN}_{k}(y)]$ when $y=x^{\prime}_{i}$. Thus, before concluding each cycle of the inner loop (lines 6-10), the $\textit{k\_bin}(k_{up})$-th entry of $hst$ associated with the sample $x^{\prime}_{ord(j)}$ is incremented.

The last step of the procedure consists in the computation of the CFOF scores. For each sample $x^{\prime}_{i}$, the associated counts are accumulated, by using the variable $count$, until their sum exceeds the value $s\varrho$ and, then, the associated value of $k_{pos}$ is employed to obtain the score. We notice that the parameter $\varrho$ is used only to determine when to stop the above computation. By using this strategy, fast-CFOF supports multi-resolution outlier analysis with no additional cost (lines 12-20).

4.2. Temporal and spatial cost

Now we consider the temporal cost of the algorithm fast-CFOF. The procedure fast-CFOF_part is executed $\left\lceil\frac{n}{s}\right\rceil$ times. During this procedure, the first part, concerning nearest neighbor counts estimation, costs $O(s\cdot(sd+s\log s+s))$, since computing distances costs $O(sd)$, sorting distances costs $O(s\log s)$, and updating counts costs $O(s)$ ($s$ iterations, each of which has cost $O(1)$). As for the second part, concerning scores computation, in the worst case all the entries of the matrix $hst$ are visited, and the cost is $O(sB)$, where $B$, the number of bins within $hst$, is a constant. Summarizing, the temporal cost of fast-CFOF is

[TABLE]

that is $O(n\cdot s\cdot\max\{d,\log s\})$. If we assume that $d$ dominates $\log s$, we can conclude that the asymptotic cost of the technique is

[TABLE]

where $s$ is independent of the number $n$ of dataset objects, and can be considered a constant depending only on $\epsilon$ and $\delta$, and $n\cdot d$ represents precisely the size of the input. We can conclude that technique is linear in the size of the input data.

Notice that the method can be easily adapted to return the top $m$ outliers, by introducing $\ell$ heaps of $m$ elements each, to be updated after each partition computation. The additional $O(n\log m)$ temporal cost has no impact on the overall cost.

As for the spatial cost, fast-CFOF_part needs $O(sB)$ space for storing counters $hst$, space $O(2s)$ for storing distances $dst$ and the ordering $ord$, and space $\ell s$ for storing scores to be returned in output, hence O(s\big{(}B+2+\ell)\big{)}. Hence, the cost is linear in the sample size. If the dataset is disk-resident, the a buffer maintaining the sample is needed requiring additional $O(sd)$ space.

4.3. Parallel fast-CFOF

The fast-CFOF algorithm can take advantage of parallelization techniques. First of all, partitions can be processed independently by different processors or nodes of a multi-processor/computer system. As for the computations pertaining to single partitions, it follows from the cost analysis that the heaviest task is the computation of pairwise distances. This task can take fully advantage of MIMD parallelism by partitioning distance computations on the cores of a multi-core processor, and of SIMD parallelization through the use of vector primitives that are part of instruction sets for computing distances. All of these computations are embarrassingly parallel, since they do not need communication of intermediate results.

Let $P$ denote the number of distinct processors/nodes available (in the multicomputer scenario the overhead due to need of partitioning the dataset on the different nodes has to be considered), let $C$ denote the number of cores (or hardware threads) per multi-core processor, and $V$ denote the width of the vector registers within each core, then the cost of the parallel version of fast-CFOF is

[TABLE]

We implemented a parallel version for multi-core processors working on disk-resident datasets that elaborates partitions sequentially, but simultaneously employs all cores and the vector registers to elaborate each single partition. Specifically, this version has been implemented by using the C Programming Language (gcc compiler), with the Advanced Vector Extensions (AVX) intrinsics functions to perform SIMD computations (vector instructions process in parallel $V=8$ pairs of float), and with the Open Multiprogramming (OpenMP) API to perform MIMD (multi-core) computations.

The implementation exploits column-major ordering of the sample points, to make the code independent of the number of attributes, and the loop unrolling and cache blocking techniques (Petersen and Arbenz, 2004): loop unrolling allows to reduce loop control instructions and branch penalties, to hide memory latencies, and to increase exploitation of replicated vectorial arithmetic logical units typical of super-scalar processors, while cache blocking improves the locality of memory accesses.

5. Experimental results

In this section, we present experimental results concerning CFOF.

Experiments are organized as follows. We discuss experimental results involving the fast-CFOF algorithm, including scalability and approximation accuracy of the approach (Section 5.1). We illustrate experiments designed to study the concentration properties of the CFOF definition on real-life data (Section 5.2). We investigate the behavior of the CFOF definition on synthetically generated multivariate data, and compare it with existing reverse nearest neighbor-based, distance-based, density-based, and angle-based outlier definitions (Section 5.3). Finally, we compare CFOF with other outlier definitions by using labelled data as ground truth (Section 5.4).

Since in some experiments, results of CFOF are compared with those obtained by other outlier detection methods, namely ODIN, AntiHub2, aKNN, LOF, and FastABOD, next we briefly recall these methods.

The aKNN method, for average KNN, is a distance-based approach that rank points on the basis of the average distance from their $k$ nearest neighbors, also called weight in the literature (Angiulli and Pizzuti, 2002, 2005; Angiulli et al., 2006). The Local Outlier Factor method (Breunig et al., 2000), LOF for short, is a density-based method which measures the degree of an object to be an outlier by comparing the density in its neighborhood with the average density in the neighborhood of its neighbors, where the density of a point is related to the distance to its $k$-th nearest neighbor.888LOF uses the concept of reachability distance to model the density of a point, which, roughly speaking, corresponds to the distance from $k$-th nearest neighbor with some minor modifications aiming to mitigate the effect of statistical fluctuations of the distance.

Differently from distance-based definitions, which declare as outliers the points where the estimated data density is low, density-based definitions score points on the basis of the degree of disagreement between the estimated density of the point and the estimated density of its surrounding or neighboring points. Thus, density-based outlier definitions are better characterized as a notion of local outlier, as opposite to distance-based definitions, representing a notion of global outlier. The Angle-Based Outlier Detection method (Kriegel et al., 2008), ABOD for short, scores data points on the basis of the variability of the angles formed by a point with each other pair of points (the score is also called ABOF, for Angle-Based Outlier Factor). The intuition is that for a point within a cluster, the angles between difference vectors to pairs of other points differ widely, while the variance of these angles will become smaller for points at the border of a cluster and for isolated points. The ABOF measures the variance over the angles between the difference vectors of a point to all pairs of other points in the dataset, weighted by the distance from these points. The FastABOD method approximates the ABOF score by considering only the pairs of points with the strongest weight in the variance, that are the $k$ nearest neighbors. In some cases, this method is not included in the comparison because of its slowness (the method has cubic temporal cost). ODIN (Hautamäki et al., 2004), also referred to as RNNc, is a reverse nearest neighbor-based approach, which uses ${\rm N}_{k}(x)$ as outlier score of $x$. Since ODIN is prone to the hubness phenomenon, the AntiHub2 method (Radovanović et al., 2015) refines ODIN by returning the weighted mean of the sum of the ${\rm N}_{k}$ scores of the neighbors of the point and of the ${\rm N}_{k}$ score of the point itself.

5.1. Experiments with fast-CFOF

Experiments of this section are designed to study the scalability (see Section 5.1.1) and the accuracy (see Section 5.1.2) of the fast-CFOF algorithm.

Experiments were performed on a PC equipped with an Intel Core i7-3635QM $2.40$GHz CPU having $4$ physical cores and $8$ hardware threads.999Ivy Bridge microarchitecture, launched in Q3’12, $6$MB of cache L3. The PC is equipped with $12$GB of main memory and the software runs under the Linux operating system.

The number $B$ of $hst$ bins was set to $1,\!000$ and the constant $c$ used to compute $k_{up}$ was set to [math]. The implementation of fast-CFOF is that described in Section 4.3 with number of processors $P=1$, number of hardware threads $C=8$, and SIMD register width $V=8$ single-precision floating point numbers ($64$ bit code having $16$ vectorial $256$ bit registers). If not otherwise stated, we assume $0.01$ as the default value for the parameters $\varrho$, $\epsilon$, and $\delta$.

The dataset employed are described next. Clust2 is a synthetic dataset family, with $n\in[10^{4},10^{6}]$ and $d\in[2,10^{3}]$, consisting of two normally distributed clusters, the first centered in the origin and having standard deviation $1$, and the second centered in $(4,\ldots,4)$ and having standard deviation $0.5$. The MNIST dataset,101010See http://yann.lecun.com/exdb/mnist/ consists of $n=60,\!000$ vectors having $d=784$ dimensions, representing handwritten digits. Digits have been size-normalized and centered in a fixed-size $28\times 28$ gray level image. The YearPredictionMSD dataset,111111See https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd or MSD for short in the following, is a subset of the Million Song Dataset, a freely-available collection of audio features and metadata for a million contemporary popular music tracks.121212See http://labrosa.ee.columbia.edu/millionsong/ The dataset consists mostly of western songs, commercial tracks ranging from $1922$ to $2011$. There are $n=515,\!345$ instances of $90$ attributes: $12$ encode timbre averages and $78$ encode timbre covariances. The SIFT10M dataset,131313See https://archive.ics.uci.edu/ml/datasets/SIFT10M consists of $11,\!164,\!866$ data points representing SIFT features ($d=128$) extracted from the Caltech-256 object category dataset.141414See http://resolver.caltech.edu/CaltechAUTHORS:CNS-TR-2007-001 Each SIFT feature is extracted from a $41\times 41$ image patch. This data set has been used for evaluating approximate nearest neighbor search methods.

Datasets are encoded by using single-precision floating point values (requiring $32$ bits) and are stored in secondary memory, where they occupy $3.8$GB ($n=10^{6}$ and $d=10^{3}$) for Clust2, $180$MB for MNIST, $177$MB for MSD, and $5.5$GB for SIFT10M.

5.1.1. Scalability of fast-CFOF

In this section, we study the scalability of fast-CFOF w.r.t. the dataset size $n$ and the dataset dimensionality $d$. For this experiment we considered the Clust2 synthetic dataset family. Figure 13a shows the execution time on Clust2 for the default sample size $s=26624$ ($\epsilon=0.01$ and $\delta=0.01$), $n\in[10^{4},10^{6}]$ and $d\in[2,10^{3}]$. The largest dataset considered is that for $n=1,\!000,\!000$ and $d=1,\!000$ whose running time was about $44$ minutes.

As for the scalability with respect to $d$, we can notice that there are no appreciable differences between the cases $d=2$ and $d=10$ and only about the ten percent of increment between the cases $d=10$ and $d=100$. Moreover, the execution time for the case $d=1,\!000$ is about a factor of two larger than that for the case $d=100$.

To understand this behavior, consider the asymptotic execution time of the parallel fast-CFOF. Since both distance computation and sorting are distributed among the CPU cores, but only distance computations are vectorized, for relatively small dimensionality values $d$, the cost of sorting distances (specifically the term $\log s$) dominates the cost of computing distances by exploiting vectorization (that is the term $\frac{d}{V}$). This confirms that the algorithm takes full advantage of the SIMD parallelism, for otherwise the cost would soon be dependent on the dimensionality $d$.151515The temporal dependence on the dimensionality could be emphasized for small $d$ values by including a vectorized sorting algorithm in the implementation of the fast-CFOF algorithm, see e.g. (Inoue and Taura, 2015), though for large $d$ values this modification should not modify the temporal trend.

Moreover, we note that the larger the dimensionality, the greater the exploitation of SIMD primitives associated with highly regular code which is efficiently pipelined (distances computation code presents higher instruction level parallelism efficiency due to reduced stalls, predictable branches, and loop unrolled and cache blocked code).

The dashed curves represent the execution times obtained by disabling the MIMD parallelism (only one thread hardware, i.e. $C=1$). The ratio between the execution time of the algorithms for $C=8$ and $C=1$ is about $7.6$, thus confirming the effectiveness of the MIMD parallelization schema. Summarizing, this experiment confirms that the parallel fast-CFOF takes full advantage of both MIMD and SIMD parallelism.

As for the scalability w.r.t. the dataset size $n$, the execution time for $n=10^{5}$ is roughly one order of magnitude larger than that for $n=10^{4}$ and the the execution time for $n=10^{6}$ is roughly two orders of magnitude larger than that for $n=10^{4}$. Moreover, the execution time for $n=10^{6}$ is about a factor of $10$ larger than that for $n=10^{5}$, which appears to be consistent with the asymptotic cost analysis.

Figure 14a shows the execution time of fast-CFOF on the datasets MNIST, MSD, MNIST10M, and Clust2 ($n=10^{6}$, $d=10^{3}$) for different sample sizes $s$, that are $s=512$ ($\epsilon=0.1$, $\delta=0.1$), $s=3,\!584$ ($\epsilon=0.025$, $\delta=0.025$), $s=15,\!360$ ($\epsilon=0.01$, $\delta=0.1$), $s=26,\!624$ ($\epsilon=0.01$, $\delta=0.01$), and $s=120,\!320$ ($\epsilon=0.005$, $\delta=0.005$). Solid lines concern the parallelized version, while dashed lines are the execution times obtained by disabling MIMD parallelism. Table 1 reports the MIMD speedup, that is ratio between the execution time of the latter version over the former version.

Except for very small sample sizes, that is up a few thousands of points, the ratio rapidly approaches the number $V=8$ of hardware threads employed. We can conclude that the overhead associated with the management of the threads is almost satisfactorily amortized starting from samples of ten thousand elements.

To understand the dependency of the execution time from the sample size, Figure 14b reports the ratio between the algorithm run time and the sample size $s$, as a function of $s$. In the range of samples considered, the above ratio is quite insensitive to the sample size and its trend can be considered approximatively constant. This observation agrees with the linear dependence on the sample size in the asymptotic cost. However, by taking a closer look, the curves are slightly deceasing till to the intermediate sample size and, then, slightly increasing. We explain this trend by noticing that, during the descending course, the MIMD speedup increases rapidly and then, during the ascending course, the cost of sorting becomes more appreciable.

5.1.2. Accuracy of fast-CFOF

The goal of this experiment is to assess the quality of the result of fast-CFOF for different sample sizes, that is different combinations of the parameters $\epsilon$ and $\delta$. With this aim we first computed the scores for a very large sample size $s_{max}=120,\!320$ associated with parameters $\epsilon=0.005$ and $\delta=0.005$. For MNIST and Clust2 ($n=100,\!000$, $d=100$) we determined the exact scores, since in these cases the sample size exceeds the dataset size. For the other two datasets, determining the exact scores is prohibitive, thus we assumed as exact scores those associated with the sample of size $s_{max}$.

Figure 15 compares the scores for $s=s_{max}$ with those obtained for the standard sample size $s=26,\!624$. The blue (smooth) curve represents the exact scores sorted in descending order and the $x$-axis represents the outlier rank position of the dataset points. As for the red (noisy) curve, it shows the approximate scores associated with the points at each rank position. The curves highlight that the ranking position tends to be preserved and that in both cases top outliers are associated with the largest scores.

To measure the accuracy of fast-CFOF, we used the Precision, or Prec for short, also referred to as $P@n$ in the literature (Craswell, 2016). The Prec measure is the precision associated with the top outliers, defined as the fraction of true outliers among the top-$\alpha$ outliers reported by the technique ($Prec@\alpha$ for short in the following). The latter measure is important in context of outlier detection evaluation, since outlier detection techniques aim at detecting the top-$\alpha$ most deviating objects of the input dataset. We also employed the Spearman’s rank correlation coefficient to assesses relationship between the two rankings (Corder and Foreman, 2014). This coefficient is high (close to $1$) when observations have a similar rank.

Tables 2, 3, 4, and 5 report the precision for $\alpha=0.001$ ($Prec@0.001$) and for $\alpha=0.01$ ($Prec@0.01$) in correspondence of different combinations of the sample size $s\in\{512,3,\!584,15,\!360,26,\!624\}$ and of the parameter $\varrho\in\{\varrho_{1},\ldots,\varrho_{5}\}$ ($\varrho_{1}=0.001$, $\varrho_{2}=0.005$, $\varrho_{3}=0.01$, $\varrho_{4}=0.05$, $\varrho_{5}=0.1$). Moreover, Table 6 reports the Spearman ranking coefficient for the same combinations of the parameters.

Both the precision and the correlation improve for increasing sample sizes $s$ and $\varrho$ values. Interestingly, excellent values can be achieved even with small samples especially for larger $\varrho$ values. Even the worst results, for $s=512$, are surprisingly good, considering also the modest cost at which they were obtained.

This behavior is remarkable. On one side, the ability to achieve very good accuracy with reduced sample sizes $s$, enables CFOF to efficiently process huge datasets. On the other side, the ability to improve accuracy by increasing $\varrho$ with a given sample size $s$, that is to say at the same temporal cost, enables CFOF to efficiently deal even large values of the parameter $\varrho$.

We point out that the latter kind of behavior has been posed as a challenge in (Radovanović et al., 2015) for the method AntiHub2. Indeed, since AntiHub2 reaches peak performance for values of $k\approx n$, they concluded that the development of an approximate version of this method represents a significant challenge since, to the best of their knowledge, current approximate $k$-nearest neighbors algorithms assume small constant $k$.

We can justify the very good accuracy of the method by noticing that the larger the CFOF score of point $x$ and, for any other point $y$, the larger the probability $p(x,y)$ that a generic dataset point will be located between $x$ and $y$ and, moreover, the smaller the impact of the error $\epsilon$ on the estimated value $\widehat{p}(x,y)$. Intuitively, the points we are interested in, that is the outliers, are precisely those less prone to bad estimations.

We point out that the trade-off between the reduction in computation time obtained by fast-CFOF and its approximation accuracy can be understood by looking both at Figures 13-14 and Tables 2-5. Before concluding, in order to make more immediately intelligible the above trade-off, we report in Figure 16 the relative execution time versus the sample size $s$ (see Figure 16a) and the $Prec@\alpha$ measure versus the relative execution time (see Figure 16b). The relative execution time is the ratio between the execution time of fast-CFOF for a given sample size $s$ and its execution time for the case $s=n$, corresponding to the temporal cost of CFOF. Since, in the case of MSD and SIFT10M, using a sample of the same size of the whole dataset resulted practically infeasible, we estimated the associated running time by exploiting interpolation.161616 We employed quadratic interpolation in order to take into account the effect of the term $\log s$ associated with sorting distances, as already discussed in Sections 4.2 and 5.1.1, obtaining an infinitesimal second order coefficient.

Figure 16a shows that the execution time of fast-CFOF corresponds to a small fraction of that required by the exact CFOF computation, and that this fraction becomes smaller and smaller as the dataset size increases, the time savings corresponding to different orders of magnitude. In Figure 16b, the solid line represents the value of $Prec@0.01$ for $\varrho=0.05$ and the dashed line the same value for $\varrho=0.01$, while the dotted line represents the ranking correlation for $\varrho=0.05$ (see also Tables 2-6). Specifically, the figure provides a picture of how the accuracy level of fast-CFOF varies with time savings. E.g., consider the solid line, the accuracy is at least close to $0.95$ even for the second sample size, corresponding to a relative execution time of a few percent on the smallest datasets and to a relative execution time smaller than the one percent on the largest ones.

5.2. Concentration on real data

In this section and in the following ones, we employed the hard-CFOF definition.

To study the concentration behavior of CFOF on real data we considered some datasets having dimensionality varying from hundreds to thousands, namely SIFT ($d=128$, $n=10,\!000$),171717The dataset consist of the base vectors of the ANN_SIFT10K vector set available at http://corpus-texmex.irisa.fr/., Isolet ($d=617$, $n=240$),181818See https://archive.ics.uci.edu/ml/datasets/isolet. MNIST test ($d=698$, $n=10,\!000$),191919See http://yann.lecun.com/exdb/mnist/. Sports ($d=5,\!625$, $n=9,\!120$),202020See https://archive.ics.uci.edu/ml/datasets/daily+and+sports+activities. Arcene ($d=10,\!000$, $n=900$),212121See https://archive.ics.uci.edu/ml/datasets/Arcene. and RNA-Seq ($d=20,\!531$, $n=801$).222222See https://archive.ics.uci.edu/ml/datasets/gene+expression+cancer+RNA-Seq. Attributes having null variance have been removed from the original datasets.

To help result interpretation, in addition to the original data, we considered also normalized and shuffled data. The normalized data is obtained by mean centering the data and then dividing by the standard deviation. The shuffled data is obtained by randomly permuting the elements within every attribute. As already noted in the literature (Francois et al., 2007), the shuffled dataset is marginally distributed as the original one, but because all the relationships between variables are destroyed, its component are independent, and its intrinsic dimension is equal to its embedding dimension.

Figures 17 and 18 show the distribution of the CFOF scores for $\varrho=0.001$ and $\varrho=0.01$, respectively, on the above described datasets. The abscissa reports score values, either in linear or in logarithmic scale, while the ordinate reports the dataset fraction, always in logarithmic scale. Scores are ranked in descending order, thus the scores on the left are associated with the outliers. Note that in the plots the dataset fraction increases exponentially from left to right.

Specifically, red curves are relative to the scores associated with the original data (solid line), the normalized data (dashed line), the shuffled data (dashed-dotted line), and the shuffled and normalized data (dotted line). To compare the empirical score distribution with the theoretical distribution described in Sections 3.3 and 3.5, we measured the kurtosis $\kappa_{orig}$ of the original data and the kurtosis $\kappa_{norm}$ of the normalized data (see Equations (2) and (3) and also Equations (15) and (5.2) in the following). Note that shuffling the data has no effect on the kurtosis. The blue solid line with circles markers is the theoretical distribution of the CFOF scores having kurtosis $\kappa_{orig}$, and the blue dashed line with circles markers is the theoretical distribution of the CFOF scores having kurtosis $\kappa_{norm}$. These curves have been obtained by leveraging the expression of Equation (5).

In general, the curves show that the distribution of the scores is unbalanced, with a small fraction of points associated with the largest scores and the majority of the points associated with score values that may vary over orders of magnitude. These curve witness for the absence of concentration on real life data.

By comparing curves for $\varrho=0.001$ and $\varrho=0.01$, it can be seen that scores increase with the parameter $\varrho$, as expected from the theoretical analysis.

Moreover, normalizing data enlarges the probability to observe larger scores. More specifically, this effect can be explained in the light of the analysis of Section 3.5, by noting that often the kurtosis of the normalized data increases sensibly. E.g. consider $\kappa_{orig}=2.2$ versus $\kappa_{norm}=24.8$ for Arcene or $\kappa_{orig}=2.2$ versus $\kappa_{norm}=895.8$ for MIST.

Consider an independent non-identically distributed random vector ${\bm{X}^{(d)}}$ having, w.l.o.g., null mean.232323The vector ${\bm{X}^{(d)}}$ can be always be replaced by the vector ${\bm{X}^{(d)}}-\mu_{{\bm{X}^{(d)}}}$. Then, according to Equations (3) and (2), it can be considered equivalent to an i.i.d. random vector having kurtosis

[TABLE]

and the kurtosis $\kappa_{orig}=\kappa_{{\bm{X}^{(d)}}}$ of the original vector is given by the ratio between the average fourth central moment and the average squared second central moment of the coordinates of ${\bm{X}^{(d)}}$.

Consider now the normalized random vector ${\bm{Z}^{(d)}}$ such that, for $i=1,\ldots,d$, $Z_{i}=X_{i}/\sigma(X_{i})$. Then, $\mu_{2}(Z_{i})=\sigma^{2}(Z_{i})=1$, and

[TABLE]

from which it can be concluded that the kurtosis $\kappa_{norm}$ of the normalized vector ${\bm{Z}^{(d)}}$ is given by the average kurtosis of the coordinates of the original vector ${\bm{X}^{(d)}}$

Thus, the kurtosis of the normalized data is larger than that of the original data when attributes have non-homogeneous kurtosis $\kappa_{X_{i}}$, with some attributes having extreme deviations, while the fourth central moments $\mu_{4}(X_{i})$ and the squared second central moments $\mu_{2}(X_{i})^{2}$ of the attributes are, on the average, much more similar.

For example, consider the dataset MNIST having $\kappa_{orig}=2.2$ and $\kappa_{norm}=895.8$. Despite its moments $\tilde{\mu}_{4}$ and $\tilde{\mu}_{2}^{2}$ are comparable, the kurtosis of the single attributes vary of four orders of magnitude (from about $1$ to $10^{4}$). This is related to the presence of a large number of zeros ($77.4\%$ of the whole dataset, after removing attributes consisting only of zeros) which are not uniformly distributed among attributes. Indeed, the attributes with very high kurtosis consist almost entirely of zeros: the attribute having kurtosis $10^{4}$ contains only one non-zero entry, while the attributes having small kurtosis contain at least the fifty percent of non-zero values. Note that is not our intention to argue that normalizing makes always sense, but only to observe and explain the behavior of the score on normalized data.

To understand the effect of normalization on the result of the CFOF technique, we computed the Spearman rank correlation between the CFOF scores associated with the original data and the CFOF scores associated with normalized data. The correlations are: $0.9511$ for SIFT, $0.7454$ for Isolet, $0.8338$ for MNIST, $0.8934$ for Sports, $0.6090$ for Arcene, and $0.8898$ for RNA-Seq. These correlations suggest that in different cases there is some agreement between the ranking induced by the method on original and normalized data.

As already noticed, the shuffled data is marginally distributed as the original data, but its component are independent. Thus, we expect that shuffled data complies better with the theoretical prediction. Indeed, the CFOF score distribution of the shuffled data follows quite closely the theoretical distribution of Equation (5). This experiment provides strong evidence for the validity of the analysis accomplished in Section 3.3. This is true both for the original shuffled data, compare the red dash-dotted curve with the solid blue line, and for the normalized shuffled data, compare the red dotted line with the blue dashed line. E.g., for Arcene the shuffled data curves appear to be superimposed with the theoretical ones. In any case, the general trend of the theoretical CFOF distribution is always followed by both the original and the normalized data.

To study the effect of varying the dimensionality, we used Principal Component Analysis to determine the principal components of the original data and, then, considered datasets of increasing dimensionality obtained by projecting the original dataset on its first $d$ principal components. Here $d$ is log-spaced between $1$ and the number of distinct principal components $d_{max}=\min\{D,n-1\}$, where $D$ denotes the number of attributes of the original dataset.

Figure 19 shows the distribution of CFOF scores for $\varrho=0.01$. The dotted (red) curve concerns the full feature space, while the other curves are relative to the projection on the first principal components. It can be seen that the general form of the score distribution holding in the full feature space is still valid for the subspace projected datasets.

In order to compare the different curves, we define a measure of abruptness of an outlier score distribution. Let $Sc$ be a set of outlier scores, let $\mbox{top}_{\alpha}(Sc)$ denote the scores associated with the top $\alpha$ outliers, and let $\mbox{med}(Sc)$ denote the median score in $Sc$. We call concentration ratio CR the ratio

[TABLE]

between the standard deviation associated with the top $\alpha$ outlier scores and the median outlier score. The numerator term above measures how well the score distribution separates the most deviating observations of the population. Since the standard deviations are relative to scores having different distributions, the term in the denominator serves the purpose of normalizing the above separation, expressing it as a quantity relative to the magnitude of the most central score. We note that the mean is not suitable for normalizing the top outlier scores, in that scores may vary of different orders of magnitude, with possibly a few large score values and a lot of negligible score values, and thus the mean would be mostly influenced by the most extreme scores.

From the analysis of Section 3.5, extremely high kurtosis distributions are those maximizing the probability to observe large scores (close to $1$), but also the ones associated with the less steep CFOF score distribution. The lower bound to the relative concentration for high kurtosis distributions can be obtained by exploiting Theorem 3.8. In this case, $\lim_{\kappa\rightarrow\infty}Pr[\mbox{CFOF}({\bm{X}^{(d)}})\leq s]=s$ and the CFOF scores are uniformly distributed between [math] and $1$. Then, $\sigma(\mbox{top}_{\alpha}(Sc))$ is the standard deviation of an uniform distribution in the interval $[1-\alpha,\alpha]$, that is $\frac{\alpha}{2\sqrt{3}}$, and $\mbox{med}(Sc)$ is the median value of an uniform distribution in the interval $[0,1]$, that is $0.5$. Hence, $\lim_{\kappa\rightarrow\infty}\mbox{CR}_{\alpha}(Sc)=\frac{\alpha}{4\sqrt{3}}$.

Figure 7 reports the concentration ratio as the dimensionality increases for both original and normalized data. For the five values of the dimensions $d_{1},d_{2},d_{3},d_{4},d_{max}$, please refer to the legends of Figures 19 and 20.

The table shows that the trend of the concentration ratio depends on the data at hand, in some cases it is monotone increasing and and some others it reaches a maximum at some intermediate dimensionality.

Unexpectedly, normalizing raises the value of the concentration ratio. Probably this can be explained by the fact that, despite the kurtosis is increased, correlations between variables are not lost, and, hence, the final effect seems to be that of emphasizing scores of the most deviating points. We note that a secondary effect of normalization is to prevent variables from having too little effect on the distance values. As pointed out in Section 3.3, Equation (5) holds also in the case of independent non-identically distributed random variables provided, that they have comparable central moments. However, since the shuffled original data is in good agreement with the theoretical prediction, normalization do not seem central to achieve comparable moments for these datasets.

Table 8 reports the concentration ratios obtained by CFOF, ODIN, AntiHub2 (aHub2, for short), aKNN, LOF, and iForest on the above datasets. ODIN presents in different cases extremely small concentration ratios. Indeed, from Equation (6), the standard deviation of the top CFOF scores tend to [math], while the median score tends to $\varrho$. In any case, CFOF has concentration ratios that are different orders of magnitude larger than that of the other scores. AntiHub2 mitigates the concentration problems of ODIN. As for the concentration ratios of distance-based and density-based methods, usually they get larger on normalized data with increased kurtosis. For example, on the normalized MNIST dataset, having a very large kurtosis, aKNN and LOF show the greatest concentration ratio. Indeed, according the analysis of Section 3.7, concentration of these scores can be avoided only for data having infinite kurtosis. iForest has very small concentration ratios on normalized data, even for large kurtosis values. The same holds for the original data, except that now the ratio appears to benefit of a larger kurtosis.

5.3. Comparison on synthetic data

This section concerns the scenario in which all the coordinates are independent.

With this aim, we generated multivariate data having the following characteristics: ($i$) independent identically distributed (i.i.d.) coordinates, ($ii$) independent non-identically distributed (i.non-i.d.) coordinates, and ($iii$) normal multivariate with non-diagonal covariance matrix.242424 In the last case, data consists of $d$-dimensional points coming from a multivariate normal distribution with covariance matrix $\Sigma=S^{\top}S$, where $S$ is a matrix of standard normally distributed random values. Since we obtained very similar results for all the above kinds of data, for the sake of shortness, in the following we report results concerning i.i.d. data.

We considered three dataset families. The first family, called Unimodal, consists of multivariate i.i.d. standard normal data.

The two other dataset families are described in (Radovanović et al., 2015) and concern the scenario involving clusters of different densities. Each dataset of this family is composed of points partitioned into two equally-sized clusters. The first (second, resp.) cluster consists of points coming from a $d$-dimensional normal distribution with independent components having mean $-1$ ($+1$, resp.) and standard deviation $0.1$ ($1$, resp.). Moreover, a variant of each dataset containing artificial outliers is obtained by moving the $\alpha$ fraction of the points maximizing the distance from their cluster center even farther from the center by $20\%$ of the distance. We refer to the family of datasets above described with (without, resp.) artificial outliers as to Multimodal artificial (Multimodal, resp.).252525We note that (Radovanović et al., 2015) experimented only the Multimodal artificial family.

We considered datasets of $n=1,\!000$ points and varied the dimensionality $d$ from $10$ to $10^{4}$. Points maximizing the distance from their cluster center were marked as outliers, and the fraction of outliers was set to $\alpha=0.05$. For each dataset we considered $N=20$ log-spaced values for the parameter $k$ ranging from $2$ to $n/2$. Results are averaged over ten runs.

To measure accuracy, we used the AUC (Area Under the ROC Curve) and the Precision, or Prec for short. The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The AUC is defined as the are under the ROC curve.

We also measured the $AUC_{max}$ and $AUC_{mean}$, defined as, respectively, the maximum and the average AUC associated with the different values of $k$ employed. Similarly, we measured the $Prec_{max}$ and $Prec_{mean}$. Within the unsupervised outlier detection scenario average performances are of interest due to the unavailability of label information that could be exploited to determine the optimal values for the parameters.

Moreover, we defined $p_{win,AUC}(Def_{1},Def_{2})$ as the fraction of configurations of the parameter $k$ for which the outlier definition $Def_{1}$ performs better than the outlier definition $Def_{2}$ in terms of AUC. By $p_{win,AUC}(Def_{1})$ we denote the fraction of configurations of the parameter $k$ for which the outlier definition $Def_{1}$ performs better than a generic outlier definition $Def_{2}$. The measures $p_{win,Prec}(Def_{1},Def_{2})$ and $p_{win,Prec}(Def_{1})$ are defined in a similar manner.

Table 9 reports the $AUC_{mean}$, $AUC_{max}$, $Prec_{mean}$, and $Prec_{max}$ obtained by CFOF, ODIN, AntiHub2, aKNN, LOF, and FastABOD on the Unimodal, Multimodal, and Multimodal artificial dataset families. The table reports also the $AUC_{max}$ an $Prec_{max}$ values obtained by iForest. We note that iForest has no neighborhood parameter and, hence, the other measure are not applicable, since they depend on the whole range of values for the parameter $k$ above described.

As for the $AUC_{mean}$ and the $Prec_{mean}$, except for the low dimensional Unimodal dataset, CFOF always obtains the best values of $AUC_{mean}$ and $Prec_{mean}$. The value associated with CFOF is most often larger than that associated with the other methods, thus suggesting that on distribution data CFOF is less sensitive to variations of the parameters than the other methods. As for the $AUC_{max}$, CFOF is able to obtain the best values in all cases. Depending on the characteristics of the dataset, also other methods are able to reach similar values. As for the $Prec_{max}$, except for Multimodal artificial which contains artificially well-separated outliers, the values obtained by CFOF are not reached by any other method.

The above conclusions are valid for all the three dataset families. Now we comment on the single families. As for the Unimodal dataset, it contains global outliers and almost all the techniques considered show good performances. However, FastABOD and, specifically, AntiHub2 sensibly worsen their quality when the dimensionality increases, while iForest tends to the random classifier ($AUC$ close to $0.5$) with a negligible precision.

As for the Multimodal and Multimodal artificial dataset, they contain local outliers and, hence, reverse nearest neighbor-based and local techniques should perform better than global techniques. The behavior of FastABOD can be explained by noticing that this technique is similar to global techniques, since the score of a point depends also on its $k$ nearest neighbors. Indeed, the value $Prec_{max}=0.5$ obtained by aKNN and FastABOD indicates these methods detect only the outliers coming from the sparsest cluster. Moreover, the AntiHub2 method needs well-separated artificial outliers to ameliorate its performances. As for iForest, its behavior on the Multimodal dataset is similar to the Unimodal case ($AUC$ close to $0.5$ and negligible precision). Moreover, even in the presence of artificially separated outliers, its accuracy remains lower than that of density-based and angle-based methods.

Figures 21, 22 show the $AUC$ (on the top left) and the Precision of the techniques for different values of the number of nearest neighbors $k$ expressed as a fraction of the dataset size, together with the $p_{win,AUC}$ and the $p_{win,Prec}$. The behavior of AntiHub2 reported in Figure 21c for $d=10,\!000$ follows that shown in (Radovanović et al., 2015) (cf. Fig. 8) for $d=100$, where the method reaches its best AUC values for $k$ ranging to about the $5\%$ to about the $50\%$ of $n$. On these kind of distribution data, CFOF shows best performances for all the measures throughout the whole range of values for the parameter $k$ and the dimensionality $d$. Remarkably, the probabilities $p_{win,AUC}(\mbox{CFOF})$ and $p_{win,Prec}(\mbox{CFOF})$ are practically always the largest.

5.4. Comparison on labelled data

In this section we considered some labelled datasets as ground truth. This kind of experiment is usually considered in the context of unsupervised outlier detection, though it must be pointed out that these of methods are not designed to take advantage from the availability of labelled data.

We considered the following ten datasets, randomly selected among those available at the UCI ML Repository262626https://archive.ics.uci.edu/ml/index.html: Breast Cancer Wisconsin Diagnostic ($n=569$, $d=32$), Epileptic Seizure Recognition ($n=11500$, $d=179$), Image segmentation ($n=2310$, $d=19$), Ionosphere ($n=351$, $d=34$), Live disorders ($n=345$, $d=7$), Ozone Level Detection ($n=2536$, $d=73$), Pima indians diabetes ($n=768$, $d=8$), QSAR biodegradation ($n=1055$, $d=41$), Wine ($n=178$, $d=13$), Yeast ($n=1484$, $d=8$),

Each class in turn is marked as normal, and a dataset composed of all the points of that class plus $10$ outlier points randomly selected from the other classes is considered. Results are averaged over $30$ runs for each choice of the normal class. For each dataset we considered $20$ distinct values for the parameter $k$ ranging from $2$ to $n/2$. Specifically, if $n>100$ then the $k$ values are log-spaced, otherwise they are linearly spaced.

We measured the $AUC_{max}$ and $AUC_{mean}$, defined as, respectively, the maximum and the average AUC associated with the different values of $k$ employed. Similarly, we measured the $Prec_{max}$ and $Prec_{mean}$ by considering the top $10$ outliers reported by each method. As already pointed out, the former measures are not applicable to iForest. Table 10 reports the above mentioned accuracy measures averaged over all the datasets, together with the probabilities $p_{win}$ that a given outlier definition reports a value for a certain accuracy measure better than another outlier definition.

As for the average accuracies, CFOF reports the largest values of $AUC_{max}$ and $Prec_{max}$, followed by LOF, while FastABOD reports the largest values of $AUC_{mean}$ and $Prec_{mean}$, followed by aKNN, though the $AUC_{mean}$ of CFOF is almost identical to that of aKNN. In addition, within the family of methods perceiving anomalies solely on the basis of their reverse nearest neighbor distribution, CFOF presents the largest average values. Since average values can be influenced by large deviations of observed AUC or $Prec$ values, the probability $p_{win}$ to obtain a better accuracy value is of interest. Remarkably, CFOF reports the greatest probability to obtain an AUC higher than any other method, both for the maximum and the mean AUC values. The probability associated with the $AUC_{max}$ of CFOF seems to indicate a superior ability to reach peak performances. Moreover, CFOF reports the greatest probability to obtain a $Prec$ higher than any other method. Also the probability concerning the mean $Prec$ is close to the best value.

In general, no correlation can be expected between the semantics underlying classes and the observed data density and, consequently, the peculiar way a certain outlier detection method perceives abnormality. In order to detect large deviations from this assumption, we exploit the Wilcoxon test. The Wilcoxon rank-sum test (Siegel, 1956) (also called the Wilcoxon-Mann-Whitney test or Mann-Whitney U test) is a nonparametric statistical test of the null hypothesis that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample.

Specifically, in the following $p_{rank,AUC}(Def_{1},Def_{2})$ represents the $p$-value from the left-tailed test, that is the probability of observing the given sets of $AUC_{max}$ values, or one more extreme, by chance if the null hypothesis that medians are equal is true against the alternative hypothesis that the median of the definition $Def_{1}$ is smaller than the median of the definition $Def_{2}$. Note that $1-p$ is the $p$-value of the test having as alternative hypothesis that the median of the definition $Def_{1}$ is greater than the median of the definition $Def_{2}$. Thus, $1-p_{rank,AUC}$ ($p_{rank,AUC}$, resp.) represents the significance level at which the hypothesis that $Def_{1}$ ($Def_{2}$, resp.) is better that $Def_{2}$ ($Def_{1}$, resp.) can be accepted. Small values of $p$-values cast doubt on the validity of the null hypothesis. Fixed a significance level $\gamma$ (usually $\gamma=0.05$) if the $p$-value is greater than $\gamma$ the null hypothesis that medians are equal cannot be rejected. Table 11 reports the $p$-values of the Wilcoxon rank-sum test.

Interestingly, the $p_{rank,AUC}$ achieved by CFOF versus any other definition ranges from $0.844$ to $0.976$. This seem to indicate that it is likely that the CFOF method will allow configurations which ranks outliers better.

Table 12 reports the $AUX_{max}$ and $Prec_{max}$ values obtained by the various methods on the considered dataset for each class label.

6. Conclusions

In this work we introduced a novel outlier definition, namely the Concentration Free Outlier Factor (CFOF). As a main contribution, we formalized the notion of concentration of outlier scores and theoretically proved that CFOF does not concentrate in the Euclidean space for any arbitrarily large dimensionality. To the best of our knowledge, there are no other proposals of outlier detection measures, and probably also of other data analysis measures related to the Euclidean distance, for which it has been provided the theoretical evidence that they are immune to the concentration effect. We also provided evidence that CFOF does not suffer of the hubness problem, since points associated with the largest scores always correspond to a small fraction of the data.

We recognized that the kurtosis of the data population is a key parameter for characterizing from the outlier detection perspective the unknown distribution underlying the data. We determined the closed form of the distribution of the CFOF scores for arbitrarily large dimensionalities and showed that the CFOF score of a point depends on its squared norm standard score and on the kurtosis of the data distribution. We theoretically proved that the CFOF score is both translation and scale-invariant and, hence, that the number of outliers coming from each cluster is directly proportional to its size and to its kurtosis, a property that we called semi–locality.

Moreover, we determined that the semi–locality is a peculiarity of reverse nearest neighbor counts: this discovery clarified the exact nature of the reverse nearest neighbor family of outlier scores. We also proved that classic distance-based and density-based outlier scores are subject to concentration both for bounded and unbounded dataset sizes, and both for fixed and variable values of the neighborhood parameter.

We showed that CFOF scores can be reliably computed by exploiting sampling techniques. Specifically, we introduced the fast-CFOF technique which does not suffer of the dimensionality curse affecting (reverse) nearest neighbor search techniques. The fast-CFOF algorithm has cost linear both in the dataset size and dimensionality, supports multi-resolution analysis, and is efficiently parallelizable. We provided a multi-core (MIMD) vectorized (SIMD) implementation. Experimental results highlight that fast-CFOF is able to achieve very good accuracy with small sample sizes, to efficiently process huge datasets, and to efficiently manage even large values of the neighborhood parameter, a property which is considered a challenge for different existing outlier methods. Moreover, experiments involving the CFOF score witness for the absence of concentration on real data and show that CFOF achieves excellent accuracy performances. The applicability of the technique is not confined to the Euclidean space or to vector spaces: it can be applied both in metric and non-metric spaces equipped with a distance function.

The CFOF technique and the properties presented in this work provide insights within the scenario of outlier detection and, more in the general, of high-dimensional data analysis. This work offers the opportunity for further investigations, including the design of algorithms with strictly bounded accuracy guarantees, the application of the technique to specific outlier detection frameworks, and many others. In particular, the suitability of the definition of being evaluated by means of sampling schemes, seems to make it appealing for the big data and data streams scenarios. In the scenario of high-dimensional data analysis, the CFOF score represents a novel notion of density measure that we believe can offer insights also in the context of other learning tasks. We are currently investigating its application in other, both unsupervised and supervised, classification contexts.