Randomized Admission Policy for Efficient Top-k and Frequency Estimation

TL;DR

This paper presents RAP, a new probabilistic algorithm for network traffic analysis that significantly reduces memory usage for frequency and top-k estimation tasks, with proven theoretical and empirical advantages.

Contribution

Introduces RAP, a novel randomized admission policy that offers substantial space savings for frequency and top-k estimation in network monitoring, supported by formal analysis and practical variants.

Findings

Space reduction by up to 128 times on heavy-tailed workloads

Memory savings between 4 and 64 times for top-k identification

Empirical and theoretical validation of RAP's efficiency

Abstract

Network management protocols often require timely and meaningful insight about per flow network traffic. This paper introduces Randomized Admission Policy (RAP) - a novel algorithm for the frequency and top-k estimation problems, which are fundamental in network monitoring. We demonstrate space reductions compared to the alternatives by a factor of up to 32 on real packet traces and up to 128 on heavy-tailed workloads. For top-k identification, RAP exhibits memory savings by a factor of between 4 and 64 depending on the skew of the workload. These empirical results are backed by formal analysis, indicating the asymptotic space improvement of our probabilistic admission approach. Additionally, we present d-Way RAP, a hardware friendly variant of RAP that empirically maintains its space and accuracy benefits.