Challenges in the capture and dissemination of measurements from high-speed networks

TL;DR

This paper discusses the multifaceted challenges of creating a large-scale high-speed network monitoring system, including technical, legal, and socio-political issues, and presents the MASTS project as a case study.

Contribution

It introduces the MASTS project for high-speed network measurement, addressing technical, legal, and socio-political challenges in data collection and dissemination.

Findings

Development of a legal framework for data anonymization and sharing

Implementation of four high-speed monitoring points in JANET network

Handling large volumes of network traffic data for research

Abstract

The production of a large-scale monitoring system for a high-speed network leads to a number of challenges. These challenges are not purely techinical but also socio-political and legal. The number of stakeholders in a such a monitoring activity is large including the network operators, the users, the equipment manufacturers and of course the monitoring researchers. The MASTS project (Measurement at All Scales in Time and Space) was created to instrument the high-speed JANET Lightpath network, and has been extended to incorporate other paths supported by JANET(UK). Challenges the project has faced have included: simple access to the network; legal issues involved in the storage and dissemination of the captured information, which may be personal; the volume of data captured and the rate at which this data appears at store. To this end the MASTS system will have established four monitoring points each capturing packets on a high speed link. Traffic header data will be continuously collected, anonymised, indexed, stored and made available to the research community. A legal framework for the capture and storage of network measurement data has been developed which allows the anonymised IP traces to be used for research pur poses.