Using a Neural Network to Detect Anomalies given an N-gram Profile

TL;DR

This paper explores using neural networks, specifically LSTM, for anomaly detection in cybersecurity, demonstrating a model that detects attacks accurately without false positives by analyzing N-gram profiles.

Contribution

It introduces and compares three neural network-based models for anomaly detection that effectively identify attack onsets with zero false positives, independent of data distribution assumptions.

Findings

Best model detected all attacks accurately

Zero false positives achieved in experiments

LSTM-based approach outperforms traditional methods

Abstract

In order to detect unknown intrusions and runtime errors of computer programs, the cyber-security community has developed various detection techniques. Anomaly detection is an approach that is designed to profile the normal runtime behavior of computer programs in order to detect intrusions and errors as anomalous deviations from the observed normal. However, normal but unobserved behavior can trigger false positives. This limitation has significantly decreased the practical viability of anomaly detection techniques. Reported approaches to this limitation span a simple alert threshold definition to distribution models for approximating all normal behavior based on the limited observation. However, each assumption or approximation poses the potential for even greater false positive rates. This paper presents our study on how to explain the presence of anomalies using a neural network, particularly Long Short-Term Memory, independent of actual data distributions. We present and compare three anomaly detection models, and report on our experience running different types of attacks on an Apache Hypertext Transfer Protocol server. We performed a comparative study, focusing on each model's ability to detect the onset of each attack while avoiding false positives resulting from unknown normal behavior. Our best-performing model detected the true onset of every attack with zero false positives.