Spark-Based Anomaly Detection: the Case of Port and Net Scan

TL;DR

This paper adapts a traditional threshold-based anomaly detection algorithm for port and net scans to Apache Spark, significantly improving execution speed and enabling real-time detection in high-speed networks, while also providing a new labeled dataset for research.

Contribution

It presents a Spark-based implementation of a simple threshold algorithm for anomaly detection, achieving higher detection performance than MAWILab and enabling real-time analysis.

Findings

Detection performance exceeds MAWILab in 95% of cases

Execution time is much shorter than trace duration

Provides a daily updated labeled dataset for research

Abstract

The two most spread network anomalies are port and net scan. In this work, we present and analyze the results obtained by traditional approaches for the detection of net scan and port scans. We use a simple threshold-based algorithm, working at flow-level and adapt it for the execution on Apache Spark. The use of Big Data Analytics technologies allows to significantly the execution times of the algorithm so to be used even in current, high-speed networks. The paper describes our approach and presents an experimental analysis in terms of detection performance and execution time. We use real traffic traces from MAWI archive and MAWILab anomaly detectors to compare with our results. The analysis shows that i) our traditional threshold-based algorithm is already able to achieve detection performance higher than MAWILab (in 95% of the considered cases with the best threshold value), currently considered the gold standard in the field; ii) the execution time is much shorter than the trace time, which makes it usable also in real time. Moreover, for each traffic trace we provide the research community with a new labeled dataset, validated by comparisons with MAWILab and extended with other anomalies not detected by it. We publish an updated dataset every day at our project website.