A Semi-Supervised Approach for Abnormal Event Prediction on Large Operational Network Time-Series Data

TL;DR

This paper introduces a semi-supervised method for predicting network anomalies using large multivariate time-series data, effectively handling data scarcity and variability in normal activities.

Contribution

The paper proposes a novel semi-supervised approach that captures dependencies in network time series and leverages limited labeled data for improved abnormal event prediction.

Findings

Outperformed state-of-the-art anomaly detection methods

Effectively used limited labeled data and abundant unlabeled data

Demonstrated strong results on real-world network logs

Abstract

Large network logs, recording multivariate time series generated from heterogeneous devices and sensors in a network, can often reveal important information about abnormal activities, such as network intrusions and device malfunctions. Existing machine learning methods for anomaly detection on multivariate time series typically assume that 1) normal sequences would have consistent behavior for training unsupervised models, or 2) require a large set of labeled normal and abnormal sequences for supervised models. However, in practice, normal network activities can demonstrate significantly varying sequence patterns (e.g., before and after rerouting partial network traffic). Also, the recorded abnormal events can be sparse. This paper presents a novel semi-supervised method that efficiently captures dependencies between network time series and across time points to generate meaningful representations of network activities for predicting abnormal events. The method can use the limited labeled data to explicitly learn separable embedding space for normal and abnormal samples and effectively leverage unlabeled data to handle training data scarcity. The experiments demonstrate that our approach significantly outperformed state-of-the-art approaches for event detection on a large real-world network log.