A compression based framework for the detection of anomalies in heterogeneous data sources

TL;DR

This paper introduces a parameter-free, compression-based framework utilizing Normalized Compression Distance and SVMs to detect security incidents across diverse cybersecurity data sources, demonstrating high flexibility and low configuration needs.

Contribution

The paper presents a novel, parameter-free methodology that applies compression-based features for anomaly detection in heterogeneous cybersecurity data sources, validated across multiple domains.

Findings

Effective in HTTP anomaly detection

Accurate spam classification

Successful tracking of domain generation algorithms

Abstract

Nowadays, information and communications technology systems are fundamental assets of our social and economical model, and thus they should be properly protected against the malicious activity of cybercriminals. Defence mechanisms are generally articulated around tools that trace and store information in several ways, the simplest one being the generation of plain text files coined as security logs. This log files are usually inspected, in a semi-automatic way, by security analysts to detect events that may affect system integrity. On this basis, we propose a parameter-free methodology to detect security incidents from structured text regardless its nature. We use the Normalized Compression Distance to obtain a set of features that can be used by a Support Vector Machine to classify events from a heterogeneous cybersecurity environment. In specific, we explore and validate the application of our methodology in four different cybersecurity domains: HTTP anomaly identification, spam detection, Domain Generation Algorithms tracking and sentiment analysis. The results obtained show the validity and flexibility of our approach in different security scenarios with a low configuration burden.