Large Scale Enrichment and Statistical Cyber Characterization of Network Traffic (Enriquecimiento a gran escala y caracterizaci\'on cibern\'etica estad\'istica del tr\'afico de red)

TL;DR

This paper presents a scalable statistical analysis framework for large-scale enriched network traffic data, revealing key cyber characteristics and activity prevalence to aid analysts.

Contribution

It introduces the use of Python GraphBLAS and PyD4M frameworks for efficient anonymized analysis of billions of network records, confirming heavy-tail distributions and activity concentration.

Findings

Most variables follow heavy-tail distributions

A small number of cyber activities dominate traffic

Enriched data enables prioritization of cyber threats

Abstract

Modern network sensors continuously produce enormous quantities of raw data that are beyond the capacity of human analysts. Cross-correlation of network sensors increases this challenge by enriching every network event with additional metadata. These large volumes of enriched network data present opportunities to statistically characterize network traffic and quickly answer a key question: "What are the primary cyber characteristics of my network data?" The Python GraphBLAS and PyD4M analysis frameworks enable anonymized statistical analysis to be performed quickly and efficiently on very large network data sets. This approach is tested using billions of anonymized network data samples from the largest Internet observatory (CAIDA Telescope) and tens of millions of anonymized records from the largest commercially available background enrichment capability (GreyNoise). The analysis confirms that most of the enriched variables follow expected heavy-tail distributions and that a large fraction of the network traffic is due to a small number of cyber activities. This information can simplify the cyber analysts' task by enabling prioritization of cyber activities based on statistical prevalence. -- Los sensores de red modernos producen enormes cantidades de datos sin procesar que est\'an m\'as all\'a de la capacidad del an\'alisis humano. Una correlaci\'on cruzada de sensores de red se convierte en un desaf\'io al enriquecer cada evento de red con metadatos adicionales. Estos grandes vol\'umenes de datos de red enriquecidos presentan una oportunidad para caracterizar estad\'isticamente el tr\'afico de red y responder a la pregunta: "?Cu\'ales son las principales caracter\'isticas cibern\'eticas de mis datos de red?" Los esquemas de an\'alisis de Python GraphBLAS y D4M permiten realizar an\'alisis estad\'isticos an\'onimos, r\'apidos y eficientes en conjuntos grandes de datos de red. Este enfoque se prueba utilizando miles de millones de muestras de datos de red an\'onimos del observatorio de Internet m\'as grande (Telescopio CAIDA) y decenas de millones de registros an\'onimos del fondo comercial con la mayor capacidad de enriquecimiento (GreyNoise). El an\'alisis confirma que la mayor\'ia de las variables enriquecidas siguen las distribuciones de cola pesada y que una gran fracci\'on del tr\'afico de red se debe a una peque\`na cantidad de actividades cibern\'eticas. Esta informaci\'on puede simplificar la tarea de los analistas cibern\'eticos al permitir la priorizaci\'on de las actividades cibern\'eticas en funci\'on de la prevalencia estad\'istica.