On Generating and Labeling Network Traffic with Realistic, Self-Propagating Malware

TL;DR

This paper introduces a method for generating realistic, labeled network traffic data by embedding defanged malware into real network environments, creating a valuable dataset for cybersecurity ML research.

Contribution

The authors present a novel approach to produce large-scale, realistic, labeled network traffic data by safely injecting malware into production networks and anonymizing the data for research use.

Findings

Generated a dataset with over 1.5 trillion connections and a petabyte of data.

Demonstrated the dataset's utility in AI/ML cybersecurity research.

Maintained high realism and security in data collection process.

Abstract

Research and development of techniques which detect or remediate malicious network activity require access to diverse, realistic, contemporary data sets containing labeled malicious connections. In the absence of such data, said techniques cannot be meaningfully trained, tested, and evaluated. Synthetically produced data containing fabricated or merged network traffic is of limited value as it is easily distinguishable from real traffic by even simple machine learning (ML) algorithms. Real network data is preferable, but while ubiquitous is broadly both sensitive and lacking in ground truth labels, limiting its utility for ML research. This paper presents a multi-faceted approach to generating a data set of labeled malicious connections embedded within anonymized network traffic collected from large production networks. Real-world malware is defanged and introduced to simulated, secured nodes within those networks to generate realistic traffic while maintaining sufficient isolation to protect real data and infrastructure. Network sensor data, including this embedded malware traffic, is collected at a network edge and anonymized for research use. Network traffic was collected and produced in accordance with the aforementioned methods at two major educational institutions. The result is a highly realistic, long term, multi-institution data set with embedded data labels spanning over 1.5 trillion connections and over a petabyte of sensor log data. The usability of this data set is demonstrated by its utility to our artificial intelligence and machine learning (AI/ML) research program.