Persistent Spread Measurement for Big Network Data Based on Register Intersection

TL;DR

This paper introduces VI-HLL, a memory-efficient architecture for persistent spread measurement in big network data, enabling accurate detection of long-term network activities with minimal memory overhead.

Contribution

The paper proposes VI-HLL, a novel compact architecture that significantly improves memory efficiency and measurement range over prior methods for persistent spread measurement.

Findings

VI-HLL outperforms V-Bitmap in memory efficiency.

It achieves accurate measurements with less than 1 bit per flow.

Theoretical and experimental results validate its effectiveness.

Abstract

Persistent spread measurement is to count the number of distinct elements that persist in each network flow for predefined time periods. It has many practical applications, including detecting long-term stealthy network activities in the background of normal-user activities, such as stealthy DDoS attack, stealthy network scan, or faked network trend, which cannot be detected by traditional flow cardinality measurement. With big network data, one challenge is to measure the persistent spreads of a massive number of flows without incurring too much memory overhead as such measurement may be performed at the line speed by network processors with fast but small on-chip memory. We propose a highly compact Virtual Intersection HyperLogLog (VI-HLL) architecture for this purpose. It achieves far better memory efficiency than the best prior work of V-Bitmap, and in the meantime drastically extends the measurement range. Theoretical analysis and extensive experiments demonstrate that VI-HLL provides good measurement accuracy even in very tight memory space of less than 1 bit per flow.

Figures

Full text

See pages 1-last of paper.pdf