Towards a Cost vs. Quality Sweet Spot for Monitoring Networks

TL;DR

This paper explores how applying signal processing techniques like the Nyquist-Shannon theorem can optimize network monitoring by reducing data collection costs while maintaining measurement quality in large datacenter networks.

Contribution

It introduces a novel approach to network monitoring cost reduction using sampling theory, demonstrating potential savings with real-world data.

Findings

Significant cost savings are possible through optimized sampling.

Sampling techniques can maintain measurement accuracy.

Practical challenges for implementation are identified.

Abstract

Continuously monitoring a wide variety of performance and fault metrics has become a crucial part of operating large-scale datacenter networks. In this work, we ask whether we can reduce the costs to monitor -- in terms of collection, storage and analysis -- by judiciously controlling how much and which measurements we collect. By positing that we can treat almost all measured signals as sampled time-series, we show that we can use signal processing techniques such as the Nyquist-Shannon theorem to avoid wasteful data collection. We show that large savings appear possible by analyzing tens of popular measurements from a production datacenter network. We also discuss the technical challenges that must be solved when applying these techniques in practice.