Time-Series Anomaly Detection Service at Microsoft

TL;DR

This paper presents a real-time time-series anomaly detection service at Microsoft, introducing a novel spectral residual and CNN-based algorithm that outperforms existing methods on multiple datasets.

Contribution

The paper introduces a new anomaly detection algorithm combining Spectral Residual and CNN, adapted from visual saliency detection, for improved accuracy and efficiency.

Findings

Achieves superior detection accuracy on public datasets.

Demonstrates effectiveness on Microsoft production data.

Outperforms state-of-the-art baselines.

Abstract

Large companies need to monitor various metrics (for example, Page Views and Revenue) of their applications and services in real time. At Microsoft, we develop a time-series anomaly detection service which helps customers to monitor the time-series continuously and alert for potential incidents on time. In this paper, we introduce the pipeline and algorithm of our anomaly detection service, which is designed to be accurate, efficient and general. The pipeline consists of three major modules, including data ingestion, experimentation platform and online compute. To tackle the problem of time-series anomaly detection, we propose a novel algorithm based on Spectral Residual (SR) and Convolutional Neural Network (CNN). Our work is the first attempt to borrow the SR model from visual saliency detection domain to time-series anomaly detection. Moreover, we innovatively combine SR and CNN together to improve the performance of SR model. Our approach achieves superior experimental results compared with state-of-the-art baselines on both public datasets and Microsoft production data.

Tables

Table 1. Table 1. Statistics of datasets

DataSet	Total Curves	Total Points	Anomaly Points
KPI	58	5922913	134114/2.26%
Yahoo	367	572966	3896/0.68%
Microsoft	372	66132	1871/2.83%

Table 2. Table 2. Result comparison of cold-start

	KPI				Yahoo				Microsoft
Model	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$
FFT	0.538	0.478	0.615	3756.63	0.291	0.202	0.517	356.56	0.349	0.812	0.218	8.38
Twitter-AD	0.330	0.411	0.276	523232.0	0.245	0.166	0.462	301601.50	0.347	0.716	0.229	6698.80
Luminol	0.417	0.306	0.650	14244.92	0.388	0.254	0.818	1071.25	0.443	0.776	0.310	16.26
SR	0.666	0.637	0.697	1427.08	0.529	0.404	0.765	43.59	0.484	0.878	0.334	2.45
SR-CNN	0.732	0.811	0.667	6805.13	0.655	0.786	0.561	279.97	0.537	0.468	0.630	25.26

Table 3. Table 3. Result comparison on test data

	KPI				Yahoo				Microsoft
Model	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$	$F_1$-score	$Precision$	$Recall$	$Time\left(s\right)$
SPOT	0.217	0.786	0.126	9097.85	0.338	0.269	0.454	2893.08	0.244	0.702	0.147	9.43
DSPOT	0.521	0.623	0.447	1634.41	0.316	0.241	0.458	339.62	0.190	0.394	0.125	1.37
DONUT	0.347	0.371	0.326	24248.13	0.026	0.013	0.825	2572.76	0.323	0.241	0.490	288.36
SR	0.622	0.647	0.598	724.02	0.563	0.451	0.747	22.71	0.440	0.814	0.301	1.55
SR-CNN	0.771	0.797	0.747	2724.33	0.652	0.816	0.542	125.37	0.507	0.441	0.595	16.13

Table 4. Table 4. Generality Comparison on Yahoo dataset

	Seasonal	Stable	Unstable	Overall	$Var$
FFT	0.446	0.370	0.301	0.364	0.060
Twitter-AD	0.397	0.924	0.438	0.466	0.268
Luminol	0.374	0.763	0.428	0.430	0.195
SPOT	0.199	0.879	0.356	0.338	0.322
DSPOT	0.211	0.485	0.379	0.316	0.120
DONUT	0.023	0.032	0.029	0.026	0.004
SR	0.558	0.601	0.556	0.563	0.023
SR-CNN	0.716	0.752	0.464	0.652	0.128

Table 5. Table 5. Features used in the supervised DNN model

Feature	Description
Transformations	Transformations to the value of each data point. We use logarithm as our transformation function and leverage the result value as a feature.
Statistics	We applied sliding windows to the time-series and treat the statistics calculated in each sliding window as features. The statistics we used include mean, exponential weighted mean, min, max, standard deviation, and the quantity of the data point values within a sliding window. We use multiple sizes of the sliding window to generate different features. The sizes are [10, 50, 100, 200, 500, 1440]
Ratios	The ratios of current point value against other statistics or transformations
Differences	The differences of current point value against other statistics or transformations

Table 6. Table 6. Train and test split of KPI dataset

DataSet	Total points	Anomaly points
Train	3004066	79554/2.65%
Test	2918847	54560/1.87%

Table 7. Table 7. Supervised results on KPI dataset

Model	$F_1$-score	Precision	Recall
DNN	0.798	0.849	0.753
SR+DNN	0.811	0.915	0.728

Equations

$$A(f)=Amplitude(\mathfrak{F}(\text{x}))$$

$$P(f)=Phrase(\mathfrak{F}(\text{x}))$$

$$L(f)=log(A(f))$$

$$AL(f)=h_q(f)\cdot L(f)$$

$$R(f)=L(f)-AL(f)$$

$$S(\text{x})=\Vert {\mathfrak{F}}^{-1}(exp(R(f)+iP(f)))\Vert$$

$$h_q(f)=\frac{1}{q^2}\left[\begin{array}{ccccc}1& 1& 1& \dots & 1\\ 1& 1& 1& \dots & 1\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& 1& 1& \dots & 1\end{array}\right]$$

$$O(x_i)=\{\begin{array}{ll}1,& \text{if S(xi)S(xi)−S(xi))>τ,}\\ 0,& \text{otherwise,}\end{array}$$

$$\overline{g}=\frac{1}{m}\sum _{i=1}^{m}g(x_n,x_{n-i})$$

$$x_{n+1}=x_{n-m+1}+\overline{g}\cdot m$$

$$x=(\overline{x}+mean)(1+var)\cdot r+x$$

Full text

anomaly detection; time-series; Spectral Residual

††journalyear: 2019††copyright: acmcopyright††conference: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 4–8, 2019; Anchorage, AK, USA††booktitle: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’19), August 4–8, 2019, Anchorage, AK, USA††price: 15.00††doi: 10.1145/3292500.3330680††isbn: 978-1-4503-6201-6/19/08††ccs: Computing methodologies Machine learning††ccs: Computing methodologies Unsupervised learning††ccs: Computing methodologies Anomaly detection††ccs: Mathematics of computing Time series analysis††ccs: Information systems Traffic analysis

1. Introduction

Anomaly detection aims to discover unexpected events or rare items in data. It is popular in many industrial applications and is an important research area in data mining. Accurate anomaly detection can trigger prompt troubleshooting, help to avoid loss in revenue, and maintain the reputation and branding for a company. For this purpose, large companies have built their own anomaly detection services to monitor their business, product and service health (Laptev et al., 2015; Vallis et al., 2014). When anomalies are detected, alerts will be sent to the operators to make timely decisions related to incidents. For instance, Yahoo releases EGADS (Laptev et al., 2015) to automatically monitor and raise alerts on millions of time-series of different Yahoo properties for various use-cases. At Microsoft, we build an anomaly detection service to monitor millions of metrics coming from Bing, Office and Azure, which enables engineers move faster in solving live site issues. In this paper, we focus on the pipeline and algorithm of our anomaly detection service specialized for time-series data.

There are many challenges in designing an industrial service for time-series anomaly detection:

Challenge 1: Lack of Labels. To provide anomaly detection services for a single business scenario, the system must process millions of time-series simultaneously. There is no easy way for users to label each time-series manually. Moreover, the data distribution of time-series is constantly changing, which requires the system recognizing the anomalies even though similar patterns have not appeared before. That makes the supervised models insufficient in the industrial scenario.

Challenge 2: Generalization. Various kinds of time-series from different business scenarios are required to be monitored. As shown in Figure 1, there are several typical categories of time-series patterns; and it is important for industrial anomaly detection services to work well on all kinds of patterns. However, existing approaches are not generalized enough for different patterns. For example, Holt winters (Chatfield, 1978) always shows poor results in (b) and (c); and Spot (Siffer et al., 2017) always shows poor results in (a). Thus, we need to find a solution of better generality.

Challenge 3: Efficiency. In business applications, a monitoring system must process millions, even billions of time-series in near real time. Especially for minute-level time-series, the anomaly detection procedure needs to be finished within limited time. Therefore, efficiency is one of the major prerequisites for online anomaly detection service. Even though the models with large time complexity are good at accuracy, they are often of little use in an online scenario.

To tackle the aforementioned problems, our goal is to develop an anomaly detection approach which is accurate, efficient and general. Traditional statistical models (Vallis et al., 2014; Siffer et al., 2017; Chatfield, 1978; Rosner, 1983; Lu and Ghorbani, 2009; Mahimkar et al., 2011; Zhang et al., 2005; Rasheed et al., 2009) can be easily adopted online, but their accuracies are not sufficient for industrial applications. Supervised models (Liu et al., 2015; Shipmon et al., 2017) are superior in accuracy, but they are insufficient in our scenario because of lacking labeled data. There are other unsupervised approaches, for instance, Luminol (lin, [n. d.]) and DONUT (Xu et al., 2018). However, these methods are either too time-consuming or parameter-sensitive. Therefore, we aim to develop a more competitive method in the unsupervised manner which favors accuracy, efficiency and generality simultaneously.

In this paper, we borrow the Spectral Residual model (Hou and Zhang, 2007) from the visual saliency detection domain to our anomaly detection application. Spectral Residual (SR) is an efficient unsupervised algorithm, which demonstrates outstanding performance and robustness in the visual saliency detection tasks. To the best of our knowledge, our work is the first attempt to borrow this idea for time-series anomaly detection. The motivation is that the time-series anomaly detection task is similar to the problem of visual saliency detection essentially. Saliency is what ”stands out” in a photo or scene, enabling our eye-brain connection to quickly (and essentially unconsciously) focus on the most important regions. Meanwhile, when anomalies appear in time-series curves, they are always the most salient part in vision.

Moreover, we propose a novel approach based on the combination of SR and CNN. CNN is a state-of-the-art method for supervised saliency detection when sufficient labeled data is available; while SR is a state-of-the-art approach in unsupervised setting. Our innovation is to unite these two models by applying CNN on the basis of SR output directly. As the problem of anomaly discrimination becomes much easier upon the output of SR model, we can train CNN through automatically generated anomalies and achieve significant performance enhancement over the original SR model. Because the anomalies used for CNN training is fully synthetic, the SR-CNN approach remains unsupervised and establishes a new state-of-the-art performance when no manually labeled data is available.

As shown in the experiments, our proposed algorithm is more accurate and general than state-of-the-art unsupervised models. Furthermore, we also apply it as an additional feature in the supervised learning model. The experimental results demonstrate that the performance can be further improved when labeled data is available; and the additional features do provide complementary information to existing anomaly detectors. Up to the date of paper submission, the $F_{1}$-score of our unsupervised and supervised approaches are both the best ever achieved on the open datasets.

The contributions of this paper are highlighted as below:

• •

For the first time in the anomaly detection field, we borrow the technique of visual saliency detection to detect anomalies in time-series data. The inspiring results prove the possibility of using computer vision technologies to solve anomaly detection problems.

• •

We combine the SR and CNN model to improve the accuracy of time-series anomaly detection. The idea is innovative and the approach outperforms current state-of-the-art methods by a large margin. Especially, the $F_{1}$-score is improved by more than 20% on Microsoft production data.

• •

From the practical perspective, the proposed solution has good generality and efficiency. It can be easily integrated with online monitoring systems to provide quick alerts for important online metrics. This technique has enabled product teams to move faster in detecting issues, save manual efforts, and accelerate the process of diagnostics.

The rest of this paper is organized as follows. First, in Section 2, we describe the details of system design, including data ingestion, experimentation platform and online compute. Then, we share our experience of real applications in Section 3 and introduce the methodology in Section 4. Experimental results are analyzed in Section 5 and related works are presented in Section 6. Finally, we conclude our work and put forward future work in Section 7.

2. system overview

The whole system consists of three major components: data ingestion, experimentation platform and online compute. Before going into more detail about these components, we will introduce the whole pipeline first. Users can register monitoring tasks by ingesting time-series to the system. Ingesting time-series from different data sources (including Azure storage, databases and online streaming data) is supported. The ingestion worker is responsible for updating each time-series according to the designated granularity, for example, minute, hour, or day. Time-series points enter the streaming pipeline through Kafka and is stored into the time-series database. Anomaly detection processor calculates the anomaly status for incoming time-series points online. In a common scenario of monitoring business metrics, users ingest a collection of time-series simultaneously. As an example, Bing team ingests the time-series representing the the usage of different markets and platforms. When incident happens, alert service combines anomalies of related time-series and sends them to users through emails and paging services. The combined anomalies show the overall status of an incident and help users to shorten the time in diagnosing issues. Figure 2 illustrates the general pipeline of the system.

2.1. Data Ingestion

Users can register a monitor task by creating a Datafeed. Each datafeed is identified by Connect String and Granularity. Connect String is used to connect user’s storage system to the anomaly detection service. Granularity indicates the update frequency of a datafeed; and the minimum granularity is one minute. An ingestion task will ingest the data points of time-series to the system according to the given granularity. For example, if a user sets minute as the granularity, ingestion module will create a task every minute to ingest a new data point. Time-series points are ingested into influxDB111https://www.influxdata.com/ and Kafka222https://kafka.apache.org/. Throughput of this module varies from 10,000 to 100,000 data points per second.

2.2. Online Compute

The online compute module processes each data point immediately after it enters the pipeline. To detect anomaly status of an incoming point, a sliding window of the time-series data points is required. Therefore, we use Flink333https://flink.apache.org/ to manage the points in memory to optimize the computation efficiency. Currently, the streaming pipeline processes more than 4 million time-series every day in production. The maximum throughput can be 4 million every minute. Anomaly detection processor detects anomalies for each single time-series. In practice, a single anomaly is not enough for users to diagnose their service efficiently. Thus, smart alert processor correlates the anomalies from difference time-series and generates an incident report accordingly. As anomaly detection is the main topic in this paper, smart alert is not discussed in more detail.

2.3. Experimentation Platform

We build an experimentation platform to evaluate the performance of anomaly detection models. Before we deploy a new model, offline experiments and online A/B tests will be conducted on the platform. Users can mark a point as anomaly or not on the portal. A labeling service is provided to human editors. Editors will first label true anomaly points of a single time-series and then label false anomaly points from anomaly detection results of a specific model. Labeled data is used to evaluate the accuracy of the anomaly detection model. We also evaluate the efficiency and generality of each model on the platform. In online experiments, we flight several datafeeds to the new model. A couple of metrics, such as click through rate of alerts, percentage of anomalies and false anomaly rate is used to decide whether the new model can be deployed to production. The experimentation platform is built on Azure machine learning service444https://azure.microsoft.com/en-us/services/machine-learning-service/. If a model is verified to be effective, the platform will expose it as a web service and host it on K8s555https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/.

3. applications

At Microsoft, it is a common need to monitor business metrics and act quickly to address the issue if there is anything outside of the normal pattern. To tackle the problem, we build a scalable system with the ability to monitor minute-level time-series from various data sources. Automated diagnostic insights are provided to assist users to resolve their issues efficiently. The service has been used by more than 200 product teams within Microsoft, across Office 365, Windows, Bing and Azure organizations, with more than 4 million time-series ingested and monitored continuously.

As an example, Michael from Bing team would like to monitor the usage of their service in the global marketplace. In the anomaly detection system, he created a new datafeed to ingest thousands of time-series, each indicating the usage of a specific market (US, UK, etc.), device (PC, windows phone, etc.) or channel (PORE, QBRE, etc.). Within 5 minutes, Michael saw the ingested time-series on the portal. At 9am, Oct-14, 2017, the time-series associated to the UK market encountered an incident. Michael was notified through E-mail alerts (as shown in Figure 3(a)) and started to investigate the problem. He opened the incident report where the top correlated time-series with anomalies are selected from a set of time-series around 9am. As shown in Figure 3(b), usage on PC devices and PORE channel can be found in the incident report. Michael brought this insight to the team and finally found that the problem was caused by a relevance issue which made users do lots of pagination requests (PORE) to get satisfactory search results.

As another example, the Outlook anti-spam team used to leverage a rule-based method to monitor the effectiveness of their spam detection system. However, this method was not easy to be maintained and usually showed bad cases on some Geo-locations. Therefore, they ingested key metrics to our anomaly detection service to monitor the effectiveness of their spam detection model across different Geo-locations. Through our API, they have integrated anomaly detection ability into the Office DevOps platform. By using this automatic detection service, they have covered more Geo-locations and received less false positive cases compared to the original rule-based solution.

4. methodology

The problem of time-series anomaly detection is defined as below.

Problem 1.

Given a sequence of real values, i.e., $\textbf{x}={x_{1},x_{2},...,x_{n}}$, the task of time-series anomaly detection is to produce an output sequence, $\textbf{y}={y_{1},y_{2},...,y_{n}}$, where $y_{i}\in\{0,1\}$ denotes whether $x_{i}$ is an anomaly point.

As emphasized in the Introduction, our challenge is to develop a general and efficient algorithm with no labeled data. Inspired by the domain of visual computing, we adopt Spectral Residual (SR) (Hou and Zhang, 2007), a simple yet powerful approach based on Fast Fourier Transform (FFT) (Van Loan, 1992). The SR approach is unsupervised and has been proved to be efficient and effective in visual saliency detection applications. We believe that the visual saliency detection and time-series anomaly detection tasks are similar essentially, because the anomaly points are usually salient in the visual perspective.

Furthermore, recent saliency detection research has shown favor to end-to-end training with Convolutional Neural Networks (CNNs) when sufficient labeled data is available (Zhao et al., 2015). Nevertheless, it is prohibitive for our application as large-scale labeled data is difficult to be collected online. As a trade-off, we propose a novel method, SR-CNN, which applies CNN on the output of SR model directly. CNN is responsible to learn a discriminate rule to replace the single threshold adopted by the original SR solution. The problem becomes much easier to learn the CNN model on SR results than on the original input sequence. Specifically, we can use artificially generated anomaly labels to train the CNN-based discriminator. In the following sub-sections, we introduce the details of SR and SR-CNN methods respectively.

4.1. SR (Spectral Residual)

The Spectral Residual (SR) algorithm consists of three major steps: (1) Fourier Transform to get the log amplitude spectrum; (2) calculation of spectral residual; and (3) Inverse Fourier Transform that transforms the sequence back to spatial domain. Mathematically, given a sequence x, we have

[TABLE]

where $\mathfrak{F}$ and $\mathfrak{F}^{-1}$ denote Fourier Transform and Inverse Fourier Transform respectively. x is the input sequence with shape $n\times 1$; $A(f)$ is the amplitude spectrum of sequence x; $P(f)$ is the corresponding phase spectrum of sequence x; $L(f)$ is the log representation of $A(f)$; and $AL(f)$ is the average spectrum of $L(f)$ which can be approximated by convoluting the input sequence by $h_{q}(f)$, where $h_{q}(f)$ is an $q\times q$ matrix defined as:

[TABLE]

$R(f)$ is the spectral residual, i.e., the log spectrum $L(f)$ subtracting the averaged log spectrum $AL(f)$. The spectral residual serves as a compressed representation of the sequence while the innovation part of the original sequence becomes more significant. At last, we transfer the sequence back to spatial domain via Inverse Fourier Transform. The result sequence $S(\textbf{x})$ is called the saliency map.

Figure 4 shows an example of the original time-series and the corresponding saliency map after SR processing. As shown in the figure, the innovation point (shown in red) in the saliency map is much more significant than that in the original input. Based on the saliency map, it is easy to leverage a simple rule to annotate the anomaly points correctly. We adopt a simple threshold $\tau$ to annote anomaly points. Given the saliency map $S(\textbf{x})$, the output sequence $O(\textbf{x})$ is computed by:

[TABLE]

where $x_{i}$ represents an arbitrary point in sequence x; $S(x_{i})$ is the corresponding point in the saliency map; and $\overline{S(x_{i})}$ is the local average of the preceding z points of $S(x_{i})$.

In practice, the FFT operation is conducted within a sliding window of the sequence. Moreover, we expect the algorithm to discover the anomaly points with low latency. That is, given a stream $x_{1},x_{2},...,x_{n}$ where $x_{n}$ is the recent point, we want to tell if $x_{n}$ is an anomaly point as soon as possible. However, the SR method works better if the target point locates in the center of the sliding window. Thus, we add several estimated points after $x_{n}$ before inputting the sequence to SR model. The value of estimated point $x_{n+1}$ is calculated by:

[TABLE]

where $g(x_{i},x_{j})$ denotes the gradient of the straight line between point $x_{i}$ and $x_{j}$; and $\overline{g}$ represents the average gradient of the preceding points. $m$ is the number of preceding points considered, and we set $m=5$ in our implementation. We find that the first estimated point plays a decisive role. Thus, we just copy $x_{n+1}$ for $\kappa$ times and add the points to the tail of the sequence.

To summarize, the SR algorithm contains only a few hyper-parameters, i.e., sliding window size $\omega$, estimated points number $\kappa$, and anomaly detection threshold $\tau$. We set them empirically and show their robustness in our experiments. Therefore, the SR algorithm is a good choice for online anomaly detection service.

4.2. SR-CNN

The original SR method utilizes a single threshold upon the saliency map to detect anomaly points, as defined in Equation (7). However, this rule is so naïve that it is natural to seek for more sophisticated decision rules. Our philosophy is to train a discriminative model on well-designed synthetic data as the anomaly detector. The synthetic data can be generated by injecting anomaly points into a collection of saliency maps that are not included in the evaluated data. The injection points are labeled as anomalies while others are labeled as normal. Concretely, we randomly select several points in the time series, calculate the injection value to replace the original point and get its saliency map. The values of anomaly points are calculated by:

[TABLE]

where $\overline{x}$ is the local average of the preceding points; $mean$ and $var$ are the mean and variance of all points within the current sliding window; and $r\sim\mathcal{N}(0,1)$ is randomly sampled.

We choose CNN as our discrimative model architecture. CNN is a commonly used supervised model for saliency detection (Zhao et al., 2015). However, as we do not have enough labeled data in our scenario, we apply CNN on the basis of saliency map instead of raw input, which makes the problem of anomaly annotation to be much easier. In practice, we collect production time-series with synthetic anomalies as training data. The advantage is that the detector can be adaptive to the change of time-series distribution, while no manually labeled data is required. In our experiments, we use totally 65 million points for training.

The architecture of SR-CNN is visualized in Figure 5. The network is composed of two 1-D convolutional layers (with filter size equals to the sliding window size $\omega$) and two fully connected layers. The channel size of the first convolutional layer is equal to $\omega$; while the channel size is doubled in the second convolutional layer. Two full connected layers are stacked before Sigmoid output. Cross entropy is adopted as the loss function; and SGD optimizer is utilized in the training process.

5. experiments

5.1. Datasets

We use three datasets to evaluate our model. KPI and Yahoo are public datasets666These two datasets are used only for research purpose and do not leveraged in production. that are commonly used for evaluating the performance of time-series anomaly detection; while Microsoft is an internal dataset collected in the production. These datasets cover time-series of different time intervals and cover a broad spectrum of time-series patterns. In these datasets, anomaly points are labeled as positive samples and normal points are labeled as negative. The statistics of these datasets are shown in Table 1.

KPI is released by AIOPS data competition (kpi, [n. d.]a, [n. d.]b). The dataset consists of multiple KPI curves with anomaly labels collected from various Internet Companies, including Sogou, Tecent, eBay, etc. Most KPI curves have an interval of 1 minute between two adjacent data points, while some of them have an interval of 5 minutes.

Yahoo is an open data set for anomaly detection released by Yahoo lab777https://yahooresearch.tumblr.com/post/114590420346/a-benchmark-dataset-for-time-series-anomaly. Part of the time-series curves is synthetic (i.e., simulated); while the other part comes from the real traffic of Yahoo services. The anomaly points in the simulated curves are algorithmically generated and those in the real-traffic curves are labeled by editors manually. The interval of all time-series is one hour.

Microsoft is a dataset obtained from our internal anomaly detection service at Microsoft. We select a collection of time-series randomly for evaluation. The selected time-series reflect different KPIs, including revenues, active users, number of pageviews, etc. The anomaly points are labeled by customers or editors manually; and the interval of these time-series is one day.

5.2. Metrics

We evaluate our model from three aspects, accuracy, efficiency and generality.

We use precision, recall and $F_{1}$-score to indicate the accuracy of our model. In real applications, the human operators do not care about the point-wise metrics. It is acceptable for an algorithm to trigger an alert for any point in a contiguous anomaly segment if the delay is not too long. Thus, we adopt the evaluation strategy888The evaluation script is available at https://github.com/iopsai/iops/tree/master/evaluation following (Xu et al., 2018). We mark the whole segment of continuous anomalies as a positive sample which means no matter how many anomalies have been detected in this segment, only one effective detection will be counted. If any point in an anomaly segment can be detected by the algorithm, and the delay of this point is no more than $k$ from the start point of the anomaly segment, we say this segment is detected correctly. Thus, all points in this segment are treated as correct, and the points outside the anomaly segments are treated as normal.

The evaluation strategy is illustrated in Figure 6. As shown in the first row of Figure 6, there are 10 contiguous points and two anomaly segments in the example time-series. The prediction results are shown in the second row. In this case, if we allow the delay as one point, i.e., $k=1$, the first segment is treated as correct and the second is treated as incorrect (because the delay is more than one point). Thus, the adjusted results are illustrated in the third row. Based on the adjusted results, the value of precision, recall and $F_{1}$-score can be calculated accordingly. In our experiments, we set $k=7$ for minutely time-series, $k=3$ for hourly time-series and $k=1$ for daily time-series following the requirement of real application.

Efficiency is another key indicator of anomaly detection models, especially for those be applied in online services. In the system, we must complete hundreds of thousands of calculations per second. The latency of the model needs to be small enough so that it won’t block the whole computation pipeline. In our experiments, we evaluate total execution time on the three datasets to compare the efficiency of different anomaly detection approaches.

Besides accuracy and efficiency, we also emphasize generality in our evaluation. As illustrated previously, an industrial anomaly detection model should have the ability to handle different types of time-series. To evaluate generality, we group the time-series in Yahoo dataset into 3 major classes (for example, seasonal, stable and unstable as shown in Figure 1) manually and compare the $F_{1}$-score on different classes separately.

5.3. SR/SR-CNN Experiment

We compare SR and SR-CNN with state-of-the-art unsupervised time-series anomaly detection methods. The baseline models include FFT (Fast Fourier Transform) (Rasheed et al., 2009), Twitter-AD (Twitter Anomaly Detection) (Vallis et al., 2014), Luminol (LinkedIn Anomaly Detection) (lin, [n. d.]), DONUT (Xu et al., 2018), SPOT and DSPOT (Siffer et al., 2017). Among these methods, FFT, Twitter-AD and Luminol do not need additional data to start, so we compare these models in a cold-start setting by treating all the time-series as test data. On the other hand, SPOT, DSPOT and DONUT need additional data to train their models. Therefore, we split the points of each time-series as two halves according to the time order. The first half is utilized for training those unsupervised models while the second half is leveraged for evaluation. Note that DONUT can leverage additional labeled data to benefit the anomaly detection performance. However, as we are aiming to get a fair comparison in the fully unsupervised setting, we do not use additional labeled data in the implementation999https://github.com/haowen-xu/donut.

The experiments are conducted in a streaming pipeline. The points of a time-series are ingested to the evaluation pipeline sequentially. In each turn, we only detect if the recent point is anomaly or not while the succeeding points are invisible. In the setting of cold-start, recommended configurations are applied to the baseline models which come from papers or codes published by the authors. For SR and SR-CNN, we set the hyper-parameters empirically. In SR, shape of $h_{q}(f)$ q is set as 3, number of local average of preceding points z is set as 21, threshold $\tau$ is set as 3, number of estimated points $\kappa$ is set as 5, and the sliding window size $\omega$ is set as 1440 on KPI, 64 on Yahoo and 30 on Microsoft. For SR-CNN, q, z, $\kappa$ and $\omega$ are set to the same value.

We report (1) $F_{1}$-score; (2) $Precision$; (3) $Recall$; and (4) CPU execution times separately for each dataset. We can see that SR significantly outperforms current state-of-the-art unsupervised models. Furthermore, SR-CNN achieves further improvement on all three datasets, which shows the advantage of replacing the single threshold by a CNN discriminator. Table 2 shows comparison results of FFT, Twitter-AD and Luminol in the cold-start scenario. We improve the $F_{1}$-score by 36.1% on KPI dataset, 68.8% on Yahoo dataset and 21.2% on Microsoft dataset compared to the best results achieved by baseline solutions. Table 3 demonstrates the comparison results of those unsupervised models which need to be trained on the first half of the dataset (labels are excluded). As shown in Table 3, the $F_{1}$-score is improved by 48.0% on KPI dataset, 92.9% on Yahoo dataset and 57.0% on Microsoft dataset than the best state-of-the-art results.

Moreover, SR is the most efficient method as indicated by the total CPU execution time in Table 2 and 3. SR-CNN achieves better accuracy with a reasonable latency increase. For generality comparison, we conduct the experiments on the second half of Yahoo dataset, which is classified into three classes manually. $F_{1}$-score on different classes of Yahoo dataset is reported separately in Table 4. SR and SR-CNN achieve outstanding results on various patterns of time-series. SR is the most stable one across the three classes. SR-CNN also demonstrates good capability of generalization.

5.4. SR+DNN

In the previous experiments, we can see that the SR model shows convincing results in the unsupervised anomaly detection scenario. However, when labels of anomalies are available, we can obtain more satisfactory results as illustrated in previous works (Liu et al., 2015). Thus, we would like to know whether our methodology contributes to the supervised scenario as well. Concretely, we treat the intermediate results of SR as an additional feature in the supervised anomaly detection model. We conduct the experiment on KPI dataset as it has been extensively studied in the AIOPS data competition (kpi, [n. d.]b).

We adopt the DNN-based supervised model (dnn, [n. d.]) which is the champion in the AIOPS data competition. The DNN architecture is composed by an input layer, an output layer and two hidden layers (shown in Figure 7). We add a dropout layer after the second hidden layer and set dropout ratio as 0.5. In addition, we apply $L_{1}=L_{2}=0.0001$ regularization to the weights of all layers. Since the output of the model indicates the likelihood of a data point being an anomaly, we search for the optimal threshold on the training set.

Each data point is associated with a feature vector, which consists of different types of features including transformations, statistics, ratios, and differences (Table 5). We follow the official train/test split of the dataset, where the statistics is shown in Table 6. We can see that the proportion of positive and negative samples is extremely imbalanced. Thus, we train our model by over-sampling anomalies to keep the positive/negative proportion to 1:2.

Experimental results are shown in Table 7. We can see that the SR feature brings 1.6% improvement in $F_{1}$-score to the vanilla DNN model. Especially, the SR-powered DNN model establishes a new state-of-the-art on the KPI dataset. To the best of our knowledge, it is the best-ever result reported on the KPI dataset up to the date of paper submission. Moreover, we draw the P-R curve of the SR+DNN and DNN methods. As illustrated in Figure 8, SR+DNN outperforms the vanilla DNN consistently on various threshold.

6. related works

6.1. Anomaly detectors

Previous works can be categorized into statistical, supervised and unsupervised approaches. In the past years, several models were subsequently proposed in the statistics literature, including hypothesis testing (Rosner, 1983), wavelet analysis (Lu and Ghorbani, 2009), SVD (Mahimkar et al., 2011) and auto-regressive integrated moving average (ARIMA) (Zhang et al., 2005). Fast Fourier Transform (FFT) (Van Loan, 1992) is another traditional method for time-series processing. For example, (Rasheed et al., 2009) highlighted the areas with high frequency change by FFT and reconfirmed it with Z-value test. In 2015, Twitter (Vallis et al., 2014) proposed a model to detect anomalies in time-series of both application metrics (e.g., Tweets Per Sec) and system metrics (e.g., CPU utilization). In 2017, SPOT and DSPOT (Siffer et al., 2017) were proposed on the basis of Extreme Value Theory (De Haan and Ferreira, 2007), the threshold of which can be selected automatically.

The performances of traditional statistical models are not satisfactory in real applications. Thus, researchers have investigated supervised models to improve the anomaly detection accuracy. Opprentice (Liu et al., 2015) outperformed other traditional detectors by using statistical detectors as feature extractors and leveraged a Random Forest classifier (Liaw et al., 2002) to detect anomalies. Yahoo EGADS (Laptev et al., 2015) utilized a collection of anomaly detection and forecasting models with an anomaly filtering layer for scalable anomaly detection on time-series data. In 2017, Google leveraged deep learning models to detect anomalies on their own dataset (Shipmon et al., 2017) and achieved promising results. However, continuous labels can not be obtained in industrial environment, which makes these supervised approaches insufficient in online applications.

As a result, advanced unsupervised approaches have been studied to tackle the problem in industrial application. In 2018, (Xu et al., 2018) proposed DONUT, an unsupervised anomaly detection method based on Variational Auto-Encoder (VAE) (Doersch, 2016). VAE was leveraged to model the reconstruction probabilities of normal time-series, while the abnormal points were reported if the reconstruction error was larger than a threshold. Besides, LinkedIn developed Luminol (lin, [n. d.]) based on (Wei et al., 2005), which segmented time-series into chunks and used the frequency of similar chunks to calculate anomaly scores.

6.2. Saliency detection approaches

Our work has been inspired by visual saliency detection models. Hou et al. (Hou and Zhang, 2007) invented the Spectral Residual (SR) model for saliency detection and demonstrated impressive performance in their experiments. They assumed that an image can be divided into redundant part and innovation part, while people’s vision is more sensitive to the innovation part. Meanwhile, the log amplitude spectrum of an image subtracting the average log amplitude spectrum captures the saliency part of the image. Guo et al. (Guo et al., 2008) argued that only phase spectrum was enough to detect the saliency part of an image and simplified the algorithm in (Hou and Zhang, 2007). Hou et al. (Hou et al., 2012) also proposed an image signature approach for highlighting sparse salient regions with theoretical proof. Although the latter two solutions showed improvement in their publications, we found that Spectral Residual (SR) was more effective in our time-series anomaly detection scenario. Moreover, supervised models based on neural networks are also used in saliency detection. For instance, Zhao et al. (Zhao et al., 2015) tackled the problem of salient object detection by a multi-context deep learning framework based on CNN architecture.

7. Conclusion & Future Work

Time-series anomaly detection is a critical module to ensure the quality of online services. An efficient, general and accurate anomaly detection system is indispensable in real applications. In this paper, we have introduced a time-series anomaly detection service at Microsoft. The service has been used by more than 200 teams within Microsoft, including Bing, Office and Azure. Anomalies are detected from 4 million time-series per minute maximally in the production. Moreover, we for the first time apply the Spectral Residual (SR) model in the time-series anomaly detection task and innovatively combine the SR and CNN model to achieve an outstanding performance. In the future, we plan to ensemble the state-of-the-art methods together to provide a more robust anomaly detection service to our customers. Besides internal serving, our time-series anomaly detection service will be published on Microsoft Azure as part of Cognitive Service101010https://azure.microsoft.com/en-us/services/cognitive-services/ shortly to external customers.