Interpretable Feature Learning in Multivariate Big Data Analysis for Network Monitoring

TL;DR

This paper extends the MBDA methodology to automatically derive interpretable features for network monitoring, enabling effective anomaly detection and diagnosis in large-scale, real-world network datasets.

Contribution

It introduces an automated feature derivation process within MBDA, enhancing interpretability and scalability for big data network analysis.

Findings

Successful detection of network anomalies in benchmark datasets

Effective diagnosis of diverse network issues

Scalable analysis with parallel processing

Abstract

There is an increasing interest in the development of new data-driven models useful to assess the performance of communication networks. For many applications, like network monitoring and troubleshooting, a data model is of little use if it cannot be interpreted by a human operator. In this paper, we present an extension of the Multivariate Big Data Analysis (MBDA) methodology, a recently proposed interpretable data analysis tool. In this extension, we propose a solution to the automatic derivation of features, a cornerstone step for the application of MBDA when the amount of data is massive. The resulting network monitoring approach allows us to detect and diagnose disparate network anomalies, with a data-analysis workflow that combines the advantages of interpretable and interactive models with the power of parallel processing. We apply the extended MBDA to two case studies: UGR'16, a benchmark flow-based real-traffic dataset for anomaly detection, and Dartmouth'18, the longest and largest Wi-Fi trace known to date.