Memory efficient distributed sliding super point cardinality estimation by GPU

TL;DR

This paper presents a GPU-based, memory-efficient distributed algorithm for real-time super point cardinality estimation under sliding time windows, improving accuracy and reducing memory use in high-speed networks.

Contribution

It introduces a novel method for recording host appearance times and two estimators for super point detection and cardinality estimation under sliding windows, enabling high accuracy with minimal memory.

Findings

Achieves highest accuracy compared to existing methods.

Uses minimal memory while maintaining performance.

Operates effectively in real-time high-speed network environments.

Abstract

Super point is a kind of special host in the network which contacts with huge of other hosts. Estimating its cardinality, the number of other hosts contacting with it, plays important roles in network management. But all of existing works focus on discrete time window super point cardinality estimation which has great latency and ignores many measuring periods. Sliding time window measures super point cardinality in a finer granularity than that of discrete time window but also more complex. This paper firstly introduces an algorithm to estimate super point cardinality under sliding time window from distributed edge routers. This algorithm's ability of sliding super point cardinality estimating comes from a novel method proposed in this paper which can record the time that a host appears. Based on this method, two sliding cardinality estimators, sliding rough estimator and sliding linear estimator, are devised for super points detection and their cardinalities estimation separately. When using these two estimators together, the algorithm consumes the smallest memory with the highest accuracy. This sliding super point cardinality algorithm can be deployed in distributed environment and acquire the global super points' cardinality by merging estimators of distributed nodes. Both of these estimators could process packets parallel which makes it becom possible to deal with high speed network in real time by GPU. Experiments on a real world traffic show that this algorithm have the highest accuracy and the smallest memory comparing with others when running under discrete time window. Under sliding time window, this algorithm also has the same performance as under discrete time window.