Hunter in the Dark: Discover Anomalous Network Activity Using Deep Ensemble Network

TL;DR

DarkHunter is a neural network-based intrusion detection system that combines supervised and unsupervised learning to improve threat detection accuracy and reduce false alarms in network security.

Contribution

It introduces a novel deep ensemble network with an unsupervised scheme to enhance detection accuracy and minimize false positives in ML-based IDSs.

Findings

Outperforms existing ML-based IDSs on UNSW-NB15 dataset.

Achieves high detection accuracy with low false positive rate.

Effectively traces threats to their source in network traffic.

Abstract

Machine learning (ML)-based intrusion detection systems (IDSs) play a critical role in discovering unknown threats in a large-scale cyberspace. They have been adopted as a mainstream hunting method in many organizations, such as financial institutes, manufacturing companies and government agencies. However, existing designs achieve a high threat detection performance at the cost of a large number of false alarms, leading to alert fatigue. To tackle this issue, in this paper, we propose a neural-network-based defense mechanism named DarkHunter. DarkHunter incorporates both supervised learning and unsupervised learning in the design. It uses a deep ensemble network (trained through supervised learning) to detect anomalous network activities and exploits an unsupervised learning-based scheme to trim off mis-detection results. For each detected threat, DarkHunter can trace to its source and present the threat in its original traffic format. Our evaluations, based on the UNSW-NB15 dataset, show that DarkHunter outperforms the existing ML-based IDSs and is able to achieve a high detection accuracy while keeping a low false positive rate.