Temporal Correlation of Internet Observatories and Outposts

TL;DR

This paper analyzes the correlation of unsolicited Internet traffic sources observed from different observatories, revealing persistent high-frequency sources and their temporal dynamics over months using advanced matrix technologies.

Contribution

It introduces a novel analysis of Internet traffic correlations between observatories using GraphBLAS and D4M technologies, and characterizes the temporal behavior of high-frequency sources.

Findings

70% of bright sources are consistently detected across observatories within 6 months.

The distribution of sources follows a Zipf-Mandelbrot distribution.

Temporal correlations fit a modified Cauchy distribution, indicating drifting high-frequency sources.

Abstract

The Internet has become a critical component of modern civilization requiring scientific exploration akin to endeavors to understand the land, sea, air, and space environments. Understanding the baseline statistical distributions of traffic are essential to the scientific understanding of the Internet. Correlating data from different Internet observatories and outposts can be a useful tool for gaining insights into these distributions. This work compares observed sources from the largest Internet telescope (the CAIDA darknet telescope) with those from a commercial outpost (the GreyNoise honeyfarm). Neither of these locations actively emit Internet traffic and provide distinct observations of unsolicited Internet traffic (primarily botnets and scanners). Newly developed GraphBLAS hyperspace matrices and D4M associative array technologies enable the efficient analysis of these data on significant scales. The CAIDA sources are well approximated by a Zipf-Mandelbrot distribution. Over a 6-month period 70\% of the brightest (highest frequency) sources in the CAIDA telescope are consistently detected by coeval observations in the GreyNoise honeyfarm. This overlap drops as the sources dim (reduce frequency) and as the time difference between the observations grows. The probability of seeing a CAIDA source is proportional to the logarithm of the brightness. The temporal correlations are well described by a modified Cauchy distribution. These observations are consistent with a correlated high frequency beam of sources that drifts on a time scale of a month.