Distributed Incast Detection in Data Center Networks

TL;DR

This paper introduces a distributed switch-level incast detection method in data centers that uses probabilistic hypothesis testing to quickly and accurately identify incast traffic from initial packets, improving detection speed and accuracy.

Contribution

The paper presents a novel distributed incast detection approach based on probabilistic hypothesis testing, addressing delays and errors in existing queue-based methods.

Findings

Significantly faster detection compared to existing methods

Higher inference accuracy in incast detection

Effective from the initial packet of flows

Abstract

Incast traffic in data centers can lead to severe performance degradation, such as packet loss and increased latency. Effectively addressing incast requires prompt and accurate detection. Existing solutions, including MA-ECN, BurstRadar and Pulser, typically rely on fixed thresholds of switch port egress queue lengths or their gradients to identify microburst caused by incast flows. However, these queue length related methods often suffer from delayed detection and high error rates. In this study, we propose a distributed incast detection method for data center networks at the switch-level, leveraging a probabilistic hypothesis test with an optimal detection threshold. By analyzing the arrival intervals of new flows, our algorithm can immediately determine if a flow is part of an incast traffic from its initial packet. The experimental results demonstrate that our method offers significant improvements over existing approaches in both detection speed and inference accuracy.