An Ensemble Approach toward Automated Variable Selection for Network Anomaly Detection

TL;DR

This paper presents an automated variable selection method for network anomaly detection that identifies core features, reducing complexity while maintaining detection performance, demonstrated on public network traffic datasets.

Contribution

The paper introduces a novel ensemble-based approach that automates variable selection by combining well-known methods, addressing challenges in stopping criteria and result variability.

Findings

Identifies a small set of core variables that approximate full-feature performance.

Demonstrates effectiveness on UNSW-NB15 and IDS2017 datasets.

Reduces feature set size without significant loss in detection accuracy.

Abstract

While variable selection is essential to optimize the learning complexity by prioritizing features, automating the selection process is preferred since it requires laborious efforts with intensive analysis otherwise. However, it is not an easy task to enable the automation due to several reasons. First, selection techniques often need a condition to terminate the reduction process, for example, by using a threshold or the number of features to stop, and searching an adequate stopping condition is highly challenging. Second, it is uncertain that the reduced variable set would work well; our preliminary experimental result shows that well-known selection techniques produce different sets of variables as a result of reduction (even with the same termination condition), and it is hard to estimate which of them would work the best in future testing. In this paper, we demonstrate the potential power of our approach to the automation of selection process that incorporates well-known selection methods identifying important variables. Our experimental results with two public network traffic data (UNSW-NB15 and IDS2017) show that our proposed method identifies a small number of core variables, with which it is possible to approximate the performance to the one with the entire variables.