Optimal Probing with Statistical Guarantees for Network Monitoring at Scale

TL;DR

This paper introduces a scalable framework for network monitoring that optimally allocates probes to estimate metrics like latency and packet loss with statistical guarantees, reducing costs while maintaining accuracy.

Contribution

It develops near-optimal, scalable algorithms based on experimental design and the Frank-Wolfe method for efficient network metric estimation at scale.

Findings

Significant reduction in probing costs compared to baselines.

Maintains low estimation errors with limited probing budgets.

Validated in real cloud network environments.

Abstract

Cloud networks are difficult to monitor because they grow rapidly and the budgets for monitoring them are limited. We propose a framework for estimating network metrics, such as latency and packet loss, with guarantees on estimation errors for a fixed monitoring budget. Our proposed algorithms produce a distribution of probes across network paths, which we then monitor; and are based on A- and E-optimal experimental designs in statistics. Unfortunately, these designs are too computationally costly to use at production scale. We propose their scalable and near-optimal approximations based on the Frank-Wolfe algorithm. We validate our approaches in simulation on real network topologies, and also using a production probing system in a real cloud network. We show major gains in reducing the probing budget compared to both production and academic baselines, while maintaining low estimation errors, even with very low probing budgets.