A Statistical Framework for Efficient Monitoring of End-to-End Network Properties

TL;DR

This paper presents a generalized statistical framework for efficiently monitoring network-wide properties by exploiting path redundancy, enabling accurate estimation and anomaly detection with fewer measurements in large-scale networks.

Contribution

The authors extend their previous framework to practical network measurements, demonstrate its robustness under network variations, and apply it to real-world problems using operational network data.

Findings

Path redundancy is robust to network changes and failures.

Small sets of measurements can accurately estimate network-wide delays.

The framework effectively detects anomalies and aids in network selection.

Abstract

Network service providers and customers are often concerned with aggregate performance measures that span multiple network paths. Unfortunately, forming such network-wide measures can be difficult, due to the issues of scale involved. In particular, the number of paths grows too rapidly with the number of endpoints to make exhaustive measurement practical. As a result, there is interest in the feasibility of methods that dramatically reduce the number of paths measured in such situations while maintaining acceptable accuracy. In previous work we proposed a statistical framework to efficiently address this problem, in the context of additive metrics such as delay and loss rate, for which the per-path metric is a sum of (possibly transformed) per-link measures. The key to our method lies in the observation and exploitation of significant redundancy in network paths (sharing of common links). In this paper we make three contributions: (1) we generalize the framework to make it more immediately applicable to network measurements encountered in practice; (2) we demonstrate that the observed path redundancy upon which our method is based is robust to variation in key network conditions and characteristics, including link failures; and (3) we show how the framework may be applied to address three practical problems of interest to network providers and customers, using data from an operating network. In particular, we show how appropriate selection of small sets of path measurements can be used to accurately estimate network-wide averages of path delays, to reliably detect network anomalies, and to effectively make a choice between alternative sub-networks, as a customer choosing between two providers or two ingress points into a provider network.