Cluster-Wide Task Slowdown Detection in Cloud System

TL;DR

This paper introduces SORN, a novel method for detecting cluster-wide task slowdowns in cloud systems by analyzing task duration distributions, addressing limitations of existing single-task anomaly detection approaches.

Contribution

The paper proposes SORN, combining a Skimming Attention mechanism and Neural Optimal Transport, to effectively detect cluster-wide slowdowns amidst fluctuations, with a new adaptive loss function for training.

Findings

SORN outperforms existing methods on real-world datasets.

The Skimming Attention effectively reconstructs compound periodicity.

The adaptive loss improves training robustness.

Abstract

Slow task detection is a critical problem in cloud operation and maintenance since it is highly related to user experience and can bring substantial liquidated damages. Most anomaly detection methods detect it from a single-task aspect. However, considering millions of concurrent tasks in large-scale cloud computing clusters, it becomes impractical and inefficient. Moreover, single-task slowdowns are very common and do not necessarily indicate a malfunction of a cluster due to its violent fluctuation nature in a virtual environment. Thus, we shift our attention to cluster-wide task slowdowns by utilizing the duration time distribution of tasks across a cluster, so that the computation complexity is not relevant to the number of tasks. The task duration time distribution often exhibits compound periodicity and local exceptional fluctuations over time. Though transformer-based methods are one of the most powerful methods to capture these time series normal variation patterns, we empirically find and theoretically explain the flaw of the standard attention mechanism in reconstructing subperiods with low amplitude when dealing with compound periodicity. To tackle these challenges, we propose SORN (i.e., Skimming Off subperiods in descending amplitude order and Reconstructing Non-slowing fluctuation), which consists of a Skimming Attention mechanism to reconstruct the compound periodicity and a Neural Optimal Transport module to distinguish cluster-wide slowdowns from other exceptional fluctuations. Furthermore, since anomalies in the training set are inevitable in a practical scenario, we propose a picky loss function, which adaptively assigns higher weights to reliable time slots in the training set. Extensive experiments demonstrate that SORN outperforms state-of-the-art methods on multiple real-world industrial datasets.