A streaming algorithm and hardware accelerator for top-K flow detection in network traffic

TL;DR

This paper presents a new streaming algorithm and FPGA hardware accelerator for accurately identifying and estimating the top-K network flows in high-speed traffic, addressing memory and performance constraints.

Contribution

It introduces a modified TowerSketch algorithm combined with a priority queue for top-K detection and implements a high-speed FPGA accelerator for real-time processing.

Findings

Achieves over 0.94 precision in top-K flow identification.

Estimates flow frequencies with less than 1.96% relative error.

Processes over 200 Gbps line rate on FPGA.

Abstract

Identifying the largest K flows in network traffic is an important task for applications such as flow scheduling and anomaly detection, which aim to improve network efficiency and security. However, accurately estimating flow frequencies is challenging due to the large number of flows and increasing network speeds. Hardware accelerators are often used in this endeavor due to their high computational power, but their limited amount of on-chip memory constrains their performance. Various sketch-based algorithms have been proposed to estimate properties of traffic such as frequency, with lower memory usage and theoretical bounds, but they often under perform with the skewed distribution of network traffic. In this work, we propose an algorithm for top-K identification using a modified TowerSketch and a priority queue array. Tested on real traffic traces, we identify the top-K flows, with K up to 32,768, with a precision of more than 0.94, and estimate their frequency with an average relative error under 1.96%. We designed and implemented an accelerator for this algorithm on an AMD VirtexU280 UltraScale+ FPGA, which processes one packet per cycle at392 MHz, reaching a minimum line rate of more than 200 Gbps.