Fast Flow Volume Estimation

TL;DR

This paper introduces fast, constant-time algorithms for network traffic volume estimation in streams and sliding windows, significantly improving speed and memory efficiency over previous methods, with practical applications to hierarchical heavy hitters.

Contribution

It presents novel algorithms for volume estimation that are faster and more memory-efficient than existing solutions, validated on real-world and synthetic data.

Findings

Up to 2.4X faster runtime for stream estimation

Over 100X memory reduction on sliding windows

2.4-7X speedup in hierarchical heavy hitter detection

Abstract

The increasing popularity of jumbo frames means growing variance in the size of packets transmitted in modern networks. Consequently, network monitoring tools must maintain explicit traffic volume statistics rather than settle for packet counting as before. We present constant time algorithms for volume estimations in streams and sliding windows, which are faster than previous work. Our solutions are formally analyzed and are extensively evaluated over multiple real-world packet traces as well as synthetic ones. For streams, we demonstrate a run-time improvement of up to 2.4X compared to the state of the art. On sliding windows, we exhibit a memory reduction of over 100X on all traces and an asymptotic runtime improvement to a constant. Finally, we apply our approach to hierarchical heavy hitters and achieve an empirical 2.4-7X speedup.