State Frequency Estimation for Anomaly Detection

TL;DR

SEQUENT is an unsupervised method that uses state visit frequency in state machines to dynamically adapt anomaly scores and improve detection of network anomalies, also providing root cause analysis.

Contribution

Introduces SEQUENT, a novel unsupervised approach leveraging state visit frequency for dynamic anomaly scoring and root cause detection in network data.

Findings

Effective in detecting network anomalies across datasets

Outperforms existing unsupervised methods in accuracy

Provides useful root cause grouping for alarms

Abstract

Many works have studied the efficacy of state machines for detecting anomalies within NetFlows. These works typically learn a model from unlabeled data and compute anomaly scores for arbitrary traces based on their likelihood of occurrence or how well they fit within the model. However, these methods do not dynamically adapt their scores based on the traces seen at test time. This becomes a problem when an adversary produces seemingly common traces in their attack, causing the model to miss the detection by assigning low anomaly scores. We propose SEQUENT, a new unsupervised approach that uses the state visit frequency of a state machine to adapt its scoring dynamically for anomaly detection. SEQUENT subsequently uses the scores to generate root causes for anomalies. These allow the grouping of alarms and simplify the analysis of anomalies. We evaluate SEQUENT's effectiveness in detecting network anomalies on three publicly available NetFlow datasets and compare its performance against various existing unsupervised anomaly detection methods. Our evaluation shows promising results for using the state visit frequency of a state machine to detect network anomalies.