Distributed super point cardinality estimation under sliding time window for high speed network

TL;DR

This paper introduces a real-time algorithm for estimating super point cardinalities in high-speed networks using a novel asynchronous timestamp counter and reversible hashing, improving accuracy and efficiency under sliding time windows.

Contribution

It proposes a new asynchronous timestamp counter and reversible hash scheme for efficient, accurate super point cardinality estimation in sliding time windows.

Findings

The algorithm detects super points in real-time network traffic.

It accurately estimates cardinalities with fewer updates.

Experiments show high effectiveness on real-world data.

Abstract

Super point is a special kind of host whose cardinality, the number of contacting hosts in a certain period, is bigger than a threshold. Super point cardinality estimation plays important roles in network field. This paper proposes a super point cardinality estimation algorithm under sliding time window. To maintain the state of previous hosts with few updating operations, a novel counter, asynchronous time stamp (AT), is proposed. For a sliding time window containing k time slices, AT only needs to be updated every k time slices at the cost of 1 more bit than a previous state-of-art counter which requires $log_2(k+1)$ bits but updates every time slice. Fewer updating operations mean that more AT could be contained to acquire higher accuracy in real-time. This paper also devises a novel reversible hash function scheme to restore super point from a pool of AT. Experiments on several real-world network traffic illustrate that the algorithm proposed in this paper could detect super points and estimate their cardinalities under sliding time window in real time.