Anomaly Detection in High Performance Computers: A Vicinity Perspective

TL;DR

This paper presents a vicinity-based statistical anomaly detection method for early failure detection in high performance computers, using anonymized system logs to improve robustness and privacy.

Contribution

It introduces a novel vicinity-based anomaly detection approach that does not require hardware adjustments or extensive data, suitable for large-scale HPC systems.

Findings

Anomaly detection precision ranges from 62% to 81%.

Method effectively detects node failures in HPC systems.

Approach preserves user and system privacy.

Abstract

In response to the demand for higher computational power, the number of computing nodes in high performance computers (HPC) increases rapidly. Exascale HPC systems are expected to arrive by 2020. With drastic increase in the number of HPC system components, it is expected to observe a sudden increase in the number of failures which, consequently, poses a threat to the continuous operation of the HPC systems. Detecting failures as early as possible and, ideally, predicting them, is a necessary step to avoid interruptions in HPC systems operation. Anomaly detection is a well-known general purpose approach for failure detection, in computing systems. The majority of existing methods are designed for specific architectures, require adjustments on the computing systems hardware and software, need excessive information, or pose a threat to users' and systems' privacy. This work proposes a node failure detection mechanism based on a vicinity-based statistical anomaly detection approach using passively collected and anonymized system log entries. Application of the proposed approach on system logs collected over 8 months indicates an anomaly detection precision between 62% to 81%.

Tables

Table 1. Table I : Hardware architecture of Taurus computing nodes

Architecture	Haswell	Sandy Bridge	Westmere	Broadwell	GPU (K20X/k80)
Node count	1456	270	180	32	108
Island	4, 5, 6	1	3	4	2

Table 2. Table II : Data sources

Data source	Data collection	Information	Granularity
System logs	Automatic	Software and hardware	Component
Outage database	Semi-automatic	Service availability	Entire system
Service notifications	Manual	Service availability	Entire system
Job status reports	Automatic	Job completion status	Node

Table 3. Table III : Sample syslog entries in their original and anonymized form

#	Original Syslog entry	Anonymized
1	(siavash) CMD (/usr/bin/check >/dev/null 2>&1)	66dc2742
2	(florina) CMD (/usr/lib32/lm/lm1 1 1)	66dc2742
3	(siavash) CMD (run-parts /etc/cron.hourly)	66dc2742
4	starting 0anacron	dd740712
5	Anacron started on 2018-01-30	e5a59462
6	Jobs will be executed sequentially	f1e7eac3
7	Normal exit (0 jobs run)	eac7924f
8	finished 0anacron	a5803a8a
9	(siavash) CMD (/usr/lib32/lm/lm1 1 1)	66dc2742
10	(root) CMD (/usr/lib32/cl/cl2 1 1)	66dc2742

Table 4. Table IV : Accuracy of the proposed failure detection method in different vicinities

Anomaly
detection	Hardware
	Resource
	Physical
	Time of
	failure
Inside vicinity	Fair	Low	High	Fair (certain failures)
Outside vicinity	No	Low	Low	No

Table 5. Table V : Results of applying the proposed failure detection method on Taurus Syslogs entries

Data format	Precision	Recall
Syslog	$62\%$	$89\%$
Anonymized Syslog	$62\%$	$89\%$
Filtered Syslog	$87\%$	$83\%$
Filtered anonymized Syslog	$81\%$	$75\%$

Full text

I Introduction

In response to the demand for higher computational power, the number of components in high performance computers (HPC) rapidly increases [1]. It is expected that Exascale HPC systems become available by 2020 [2]. Besides increasing the quantity of computing resources, achieving high performance is also dependent on the optimized utilization of available resources, as well as on the continuous operation of the HPC system as a whole. Over the past decades, scientists proposed various methods and algorithms to adjust the workload on HPC systems to achieve the highest possible performance. With the drastic increase in the number of HPC system components, it is expected to observe a sudden increase in the number of failures which consequently poses a threat to the continuous operation of the HPC systems [3]. Therefore, detecting failures as early as possible and, ideally, predicting them, is a necessary step to avoid interruptions in continuous HPC systems operation.

A failure in general is an (observed) incorrect behavior with respect to the expected behavior. Failures can be observed and analyzed at different granularities, from a single transistor to an entire HPC system. Nodes are the smallest units in HPC systems that have a fully functional computational software stack and can be added or removed from HPC systems with minimum side-effects on the other computing units. Therefore, the granularity of failure detection in this work is set at the node level. Heterogeneity of computing nodes, as well as the topology of HPC systems, are other effective factors that influence the overall system performance. In accordance with node-related effective factors such as physical location, role in the cluster, computational workload, hardware properties, and so forth, various categorizations of computing nodes are possible. Hereafter, the term vicinity is used to refer to nodes that exhibit similar properties such as the ones mentioned above. The concept of vicinity defines new dimensions of node correlation, beyond the natural temporal and spatial correlations. Subsection II-A provides examples and describes the vicinity of the nodes in more detail.

An anomaly is an (observed) unexpected behavior with respect to the expected behavior. In contrast to failures, unexpected behaviors are not necessarily incorrect behaviors. Anomaly detection is a well-known general purpose approach for detecting failures in computing systems [4]. In HPC systems, system log analysis can be used for anomaly detection for the purpose of preventing failures [5, 6, 7, 8]. All HPC systems on the current TOP500 [9] list are Linux-based. Therefore, they all generate system log (Syslog) [10] messages by default. The goal of this work is to detect node failures via analyzing fully anonymized system logs using a vicinity-based statistical anomaly detection approach. The use-case of this study is a production Petascale HPC system, called Taurus111https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/SystemTaurus.

In addition to technical data, system log entries contain sensitive data about the system and its users. Therefore, addressing the data privacy [11] concerns is a fundamental requirement of any mechanism that detects anomalies on production HPC systems via system log analysis. The anonymization of system log entries, prior to performing Syslog analysis while retaining useful data, addresses the data privacy concerns [12].

In this work, the Taurus nodes are first categorized into four different vicinities based on similarities they exhibit in terms of (1) hardware architecture, (2) resource allocation, (3) physical location, and (4) time of failures. Then, the anomalies are detected within each vicinity. Subsequently, the effectiveness of performing anomaly detection in each vicinity for the purpose of failure detection is compared and discussed. To assess the usefulness of the proposed method of anomaly detection on anonymized data, a copy of 8 months of Taurus system logs was anonymized via the PaRS [13] anonymization mechanism and the anonymized system logs were used as the input data.

The main contributions of this work are: (1) proposing a node failure detection mechanism via a vicinity-based statistical anomaly detection approach that employs a passive data collection approach, as well as (2) analyzing the effectiveness of anomaly detection method in various vicinities using the 8 months of Taurus HPC cluster system logs. In addition, (3) to the best of our knowledge this is the first work on anomaly detection that is capable of utilizing both original and anonymized Syslog entries with a high degree of equivalence between the analysis results.

The remainder of this work is organized as follows. The node vicinities, data sources, and anonymization method are introduced in Subsections II-A, II-B, and II-C respectively. The method to identify node failures employed in this work is described in Subsection II-D. The proposed anomaly detection method is described in detail in Subsection II-E. The impact of anomaly detection in different node vicinities is analyzed in Section III. The background and current related works are introduced in Section IV. Finally, the work is concluded and future work directions are discussed in Section V.

II Proposed Anomaly Detection Methodology

Taurus is an HPC cluster composed of 2,046 computing nodes. The computing nodes are organized in 6 islands based on their hardware properties222Detailed hardware information of Taurus: https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/HardwareTaurus. Figure 1 provides a schematic illustration of the Taurus node topology. Each letter represents a single computing node. Nodes with identical colors are of identical hardware (processing unit) architecture.

II-A Vicinities

Computing nodes with similar characteristics are considered to be in the vicinity of each other. Node characteristics include any physical, spatial, temporal, or logical properties of the computing nodes. A group of computing nodes located in the same rack, performing different tasks of the same job, or sharing a common resource (e.g., file system, power supply unit), are all examples of nodes in the vicinity of each other.

The concept of vicinity defines new dimensions of node correlation, beyond the natural temporal and spatial correlations. Each vicinity can be imagined as a new dimension in which two separated entities (nodes) become correlated. For example, points $A:(1,10)$ and $B:(4,6)$ in a 2D Cartesian representation are separated by a distance of $4-1=3$ on the $X$ axis and $10-6=4$ on the $Y$ axis, respectively. Defining the new dimension $Z$, according to a common (but so far unseen) feature of $A$ and $B$ would result in a 3D representation of $A:(1,10,5)$ and $B:(4,6,5)$. Here ’$5$’ denotes that common feature. In the new 3D representation, even though $A$ and $B$ are still separated on $X$ and $Y$, their distance on the dimension Z will be $5-5=0$. In another word, $A$ and $B$ will be in the vicinity of each other from the $Z$ axis perspective.

In this work, node vicinities are observed from four different perspectives: (1)\leavevmode\nobreak\hardware architecture, (2)\leavevmode\nobreak\resource allocation, (3)\leavevmode\nobreak\physical location, and (4)\leavevmode\nobreak\time of failure. The first perspective denotes a node vicinity according to the node’s physical properties, the second perspective emphasizes the node’s logical properties, while the third and fourth perspectives denote spatial and temporal properties, respectively. All other correlations among nodes can be mapped onto these four proposed vicinities, e.g., nodes connected to a single switch can be mapped onto the physical location vicinity. In Subsections II-A1, II-A2, II-A3, and II-A4 these four vicinities are explained in more detail, based on the Taurus architecture.

The node vicinities are intended to mitigate the major characteristic differences between nodes. Therefore, in cases that several parameters influence a certain node’s characteristic, the most dominant parameter is considered to identify the node’s vicinity. All nodes in island 2 beside their Sandy bridge or Haswell CPUs are equipped with graphical processing units (GPU). Since the majority of jobs submitted to island 2 mainly utilize GPUs rather than CPUs, GPUs are considered as dominant processing units of these nodes. Therefore, in this work, island 2 is considered as a homogeneous GPU island, despite the heterogeneity of its nodes’ CPUs.

It is important to emphasize that in the context of this work, two nodes in the vicinity of each other are not necessarily physically co-located. In fact, they may even belong to physically separated partitions of the HPC system.

II-A1 Hardware Architecture Vicinity

Computing nodes on Taurus may be of 4 different processors architectures: Intel Haswell, Broadwell, Sandy Bridge, and Westmere. 108 nodes with Sandy Bridge and Haswell processors are also equipped with GPUs (NVIDIA Tesla K20X and K80). According to their hardware architectures, the 2,046 computing nodes on Taurus can be divided into 5 categories. The node’s dominant processor architecture and the number of nodes in each architecture category are shown in Table I. A schematic illustration of the Taurus topology, including the type of each node’s hardware architecture is provided in Figure 1. Nodes with identical colors in Figure 1 are in the vicinity of each other from the hardware architecture perspective.

II-A2 Resource Allocation Vicinity

Taurus employs Slurm [14] for job scheduling. The submitted jobs are allocated resources based on direct requests of the users, system policies, and the status of available resources. All nodes that execute tasks of the same job are in the vicinity of each other from the resource allocation perspective. In contrast to the static nature of the hardware architecture vicinity perspective, the resource allocation vicinity is fully dynamic and may change frequently as the running jobs are completed and new jobs are submitted to the cluster.

II-A3 Physical Location Vicinity

Various granularities can be used to express the physical location of a node in Taurus, e.g., chassis, rack, or island. Since the power, temperature, and connectivity of all nodes located in a single rack are controlled together, this work considers racks as the physical location granularity. Each row of an island shown in Figure 1 represents one rack of nodes. All nodes located in the same rack are in the vicinity of each other from the physical location perspective.

II-A4 Time of Failure Vicinity

Often failure is a consequence of various node-level events on and of properties of several nodes. However, a failure in itself is observable on a particular node at a specific moment. Therefore, the time of failure is considered as a temporal property of that particular node even though, several nodes may fail due to the same reason. From this perspective, all nodes that experience a failure within the same predefined time interval, fall into the same vicinity category. In this work, the time of failure interval considered is 10 minutes. The 10-minute time interval is chosen according to the results of the previous study on Taurus failure correlations [8]. That study revealed that the majority of failures correlated on Taurus occurred within 10 minutes of each other. Therefore, failures that occur across the entire system within 10 minutes of each other are assumed to be in the same temporal vicinity. Thus, the nodes on which such failures occur are in the vicinity of each other from the time of failure perspective.

II-B Data Sources

System logs represent the main data source in this work. Syslogs of Taurus were collected for a period of one year from January to December 2017. Syslog daemons on each node collected the system log entries and forwarded them to a central log collector. To maintain the neutrality of the results and to provide a general approach, the pre-configured Syslog daemons were used without any changes, except for being configured to forward Syslog entries to the central log collector.

In addition to Syslog entries, three other data sources were considered to improve the accuracy of the failure identification method: outage database, service notifications, and job status reports. The outage database reports all system outages from the users’ perspective, the service notifications notify users regarding future scheduled maintenances and system-wide outages, and the job status reports indicate the final status of a job after its execution. Table II provides an overview of the four data sources used in this work.

There are certain data gaps in the four data sources used in this work. The gaps are mainly incurred due to the interruption of the data collection mechanism. The Syslog entries cover the full period of one year from January to December 2017. The job status reports generated by Slurm showed in Figure 2 covers the period of 28-02-2017 to 14-11-2017. The service notifications and the outage database, provide information from a higher perspective and are available for the whole period of one year from January to December 2017.

Given these available data, the focus of this work is on the period of 01-03-2017 to 31-10-2017. The existence of certain gaps in data sources is reportedly a common challenge in similar studies [15].

II-C Anonymization of System Logs

System log entries contain sensitive data about the system and its users. Therefore, addressing data privacy concerns is a fundamental requirement of any mechanism that performs system log analysis. The anonymization of system log entries, prior to performing Syslog analysis, addresses the data privacy concerns.

PaRS [13] is an anonymization mechanism that provides full anonymization through hashing the message part of Syslog entries. PaRS substitutes all variables and sensitive terms in each log entry with constant values and maps each substituted entry to a hash key. Through anonymization, the partial semantic of log entries required for anomaly detection via Syslog analysis is preserved. The rightmost column in Table III contains the anonymized Syslog entries which correspond to the raw log entries shown in the middle column (prior to anonymization). As shown in Table III, an identical hash key is generated for entries with similar raw log semantics.

II-D Taurus Node Failures

To assess the proposed method’s functionality, all Taurus node failures must be known. Due to various technical reasons, the complete list of all node failures on Taurus for the period of this work is not available. This step is aimed to provide a complete list of Taurus node failures as the ground truth for further analysis and comparisons in Section III.

Node failures in computing systems can be divided into two main categories according to their root causes: (1) Failures that occur during the normal operation of the HPC system caused by internal factors, such as software and hardware errors or race conditions (i.e., regular failures). (2) Failures that occur due to external factors, such as power outages and human errors. Analyzing the impact of the external causes of node failures requires additional information regarding external factors that are not available, e.g., detailed information about the behavior of the HPC system power supply. Therefore, the focus in this work is on the first group of node failures namely regular failures, typically caused by internal factors. The next step to identify such failures is to detect node outages and to distinguish regular failures from those which may occur as a result of external factors, such as maintenance, human errors, and others. The failure detection workflow in Taurus is shown in Figure 4.

Computing nodes of Taurus generate and send Syslog entries to a central log collector which stores them for future analysis. This is a passive log collection mechanism chosen due to imposing no additional overhead, and to be applicable to other HPC systems. However, the failure identification process becomes more challenging in comparison to the use of active log collection mechanisms. Due to using the passive log collection mechanism, a node outage can be confidently detected only when a direct indication in the form of a log entry is generated by the failing node and correctly received and stored by the central log collector, e.g.,” Kernel panic - not syncing: Fatal Exception.” However, in many cases, a node outage leaves no direct indication in system logs. A workaround is to assume the absence of log entries for more than a certain period of time as an indication of a potential outage. Nonetheless, this assumption is not accurate. For various reasons such as CPU overloading or network congestion, the flow of system log entries from the computing nodes to the central log collector may be interrupted or delayed, which could be assumed as an outage, even though the computing nodes are functional. Also, in many cases immediately after the occurrence of an outage, the protection mechanisms recover the node. In both latter scenarios (temporary interruption in data flow and automatic node recovery), an active node probing approach may also fail to correctly detect all node outages.

Analyzing Taurus system logs revealed that during a healthy boot all nodes leave similar footprints in Syslog entries. When a node fails to generate the expected footprint at boot time, it is an indication of a faulty boot process and thus the node will be either automatically rebooted or it will fail shortly thereafter. The higher frequency of log generation at boot time in comparison with the normal operation is another indicator of a boot event, which can be used to identify even a problematic boot process.

The proposed node outage detection method in this work first detects the node boot events. Afterward, Syslog entries are backtracked until the last Syslog entry before the boot event is found. The timestamp of the last Syslog entry prior to the boot event is considered as the point of the outage. All node outages will be identified using the proposed method. The only exception is when a node fails and has no further successful boot process. In such cases, comparing the timestamp of the last available Syslog entry with the current time (i.e., $31$-$10$-$2017$ $23$:$59$:$59$) reveals the missing node outages. Figure 3 illustrates all detected node outages on Taurus over the course of 2017.

The detected outages are compared against the information from other available data sources (mentioned in Table II). When a node outage occurs outside of the scheduled maintenance period and no job could be accomplished on that particular node at the time of the detected outage, the detected outage represents a regular failure. As Figure 2 illustrates, it is common that certain jobs on a specific node fail, although other jobs on the same node are accomplished simultaneously. Also, when a node outage is recorded in the outages database that monitors the availability of the HPC system from the users’ perspective, it is considered as a regular failure.

II-E Anomaly Detection

The common behavior of most nodes within a node vicinity is considered as normal behavior in that vicinity. A node’s behavior is defined as the Syslog Generation frequency of the node (hereafter SG). The SG parameter is dynamically calculated based on the number of Syslog entries received from a computing node during a fixed time window (e.g. 30 minutes) prior to the current (observation) moment. The SG parameter of each node is compared against the SG of other nodes in the same vicinity. Based on these comparisons, the normal value of the SG parameter of certain nodes at a given moment in time is calculated. Once the deviation of a node’s SG parameter from the normal behavior exceeds a certain threshold, the node’s behavior is considered abnormal. The deviation threshold is dynamically calculated within each vicinity333A sample code written in python to demonstrate the calculation of dynamic thresholds via k-means is available: ghiasvand.net/u/param. To calculate the deviation threshold, all nodes within a vicinity (i.e. one row of Figure 5) are partitioned into 2 clusters based on their SG parameter via a clustering method such as K-Means. The deviation threshold is the relative density of resulting clusters which is calculated as the sum of squared distances of samples to their closest cluster center444Also known as within cluster sum..

Figure 5 illustrates the behavior of node 1110 and 8 other neighboring nodes (physically located next to each other) prior to a point in time when 11 failures occurred on node 1110 in the year 2017. Figure 6 on the other hand, illustrates the behavior of the same nodes prior to 11 random points of time in which node 1110 functioned normally. In Figure 5 and Figure 6, the timestamp at the beginning of each row represents the observation moment in the year 2017. The value of each cell represents the SG parameter of the respective node (column’s header) within a time interval of 30 minutes prior to the observation moment. Cells with abnormal behavior are shown in orange. The cell coloring in each row is relative to the value of other cells in that particular vicinity (row). According to Figure 5, node 1110 experienced 11 failures in 2017. For 7 out of the 11 failures illustrated in Figure 5, the deviation of the SG parameter correctly identifies the abnormal behavior of node 1110.

In the example provided in Figure 5, the SG parameters were obtained for the nodes physical location vicinity. The same comparisons were made within other node vicinities. In Section III the effectiveness of the proposed anomaly detection method in each node vicinity is discussed.

III Impact of Vicinities on Anomaly Detection

The proposed anomaly detection method was applied to 8 months of Taurus system logs. Regular failures were detected based on the node vicinities introduced in Subsection II-A. The proposed method is applicable to hardware architecture, resource allocation, and physical location vicinities for the purpose of online anomaly detection. However, the time of failure vicinity can only be used as an offline approach to analyze the nodes’ behavior after each failure occurrence.

Results of the proposed method’s application on Taurus system logs were compared against the set of regular failures detected in Subsection II-D. The following subsections describe the impact of performing the proposed anomaly detection method in each vicinity in more detail.

III-A Impact of Hardware Architecture Vicinity

Taurus nodes are located in 6 islands. As shown in Figure 1, island 4 hosts nodes with different processor types, while islands 1, 2, 3, 5, and 6 are homogeneous. Although the nodes’ hardware architecture influences the job allocation, as Figure 2 illustrates there is no noticeable difference among job allocation patterns on various Taurus islands. However, as shown in Figure 3, with the exception of islands 5 and 6 -which comprise of identical processor types- the node outages have different distribution patterns on each island.

Figure 7 illustrates a one-to-one comparison of Syslog generation patterns in all 6 islands of Taurus. This figure visualizes the temporal and spatial patterns among more than 46K, 82K, 45K, 968K, 940K, and 1M Syslog entries generated by islands 1 to 6, respectively. Islands 5 and 6 present an almost identical pattern, which is also very similar to island 4. In contrast, the system log generation pattern on each of the islands 1, 2, and 3 has a completely different pattern.

The comparison shown in Figure 7 indicates that the processor architecture has a direct impact on node behavior. Therefore, the behavior of nodes in island 1 (Sandy Bridge) should not be predicted based on the behavior of nodes in island 3 (Westmere), while a similar behavior is expected from nodes in island 5 (Haswell) and island 6 (Haswell).

No additional patterns were detected when conducting a similar analysis based on the amount of node’s physical memory. The use of the proposed anomaly detection method on nodes with different hardware architecture proved to be virtually impossible. Detecting anomalies within the hardware architecture vicinity on Taurus also revealed several false positives.

It is intended to improve the accuracy of the proposed statistical anomaly detection method by identifying the most relevant vicinity perspective among the four vicinities considered in this work. The proposed method detects anomalies by analyzing fully anonymized system logs. To the best of our knowledge, there is no similar approach for detecting anomalies using fully anonymized system logs. Therefore, a quantitative comparison cannot be conducted. However, Subsection III-E shows a qualitative comparison of the proposed method’s accuracy inside and outside of each vicinity.

III-B Impact of Resource Allocation Vicinity

Figure 2 illustrates jobs that failed due to node failures, as reported by Slurm. Several jobs which were allocated on multiple nodes were terminated due to node failures during the 8-month period considered in this work. However, except for one incident in which a misbehaving job was causing the node failures, no direct correlation between the node failures and resource allocation was found. Therefore, among a group of computing nodes executing various tasks of the same job, as long as the running job is not causing a node failure itself, the probability of a second node failure occurrence is not higher than on the rest of the cluster.

It is important to emphasize that the globally shared resources are not included in this analysis and conclusion. The abnormal behavior of globally shared resources such as the distributed file system may cause node failures itself. Globally shared resources are, however, unique entities in the cluster. Thus, directly including them in anomaly detection will not bring any additional benefit. Globally shared resources are, in fact, indirectly considered in anomaly detection as correlations between nodes.

The use of the proposed anomaly detection method on resource allocation vicinity did not improve the accuracy of anomaly detection. However, the low number of failed jobs allocated on multiple nodes prevents drawing a robust conclusion.

III-C Impact of Physical Location Vicinity

Computing nodes are located inside HPC systems according to their function and architecture. Therefore, the physical location of the nodes is not an independent factor. The physical location of each computing node on Taurus can uniquely be addressed by its island, rack and chassis number. Analyzing the distribution of failures on Taurus as shown in Figures 2 and 3, reveals a strong correlation among failures in each rack. Therefore, in comparison with the rest of the cluster, it is more likely to observe the second failure in a rack that already experienced a node failure.

Out of the 2,046 computing nodes on Taurus, about $20\%$ of nodes never reported a node failure, and $60\%$ of computing nodes experienced 10 or fewer node failures during the time period considered in this work. The remaining $20\%$ of computing nodes are responsible for more than $70\%$ of all node failures. As shown in Figure 8, the probability of a future node failure is in proportion to the number of previous node failures.

Application of the proposed anomaly detection method on the physical location vicinity provided the most accurate results among the three vicinities of hardware architecture, resource allocation, and physical location. The higher accuracy and the static nature of physical location vicinity make it a good candidate for the application of the proposed anomaly detection method.

III-D Impact of Time of Failure Vicinity

The time of failure vicinity includes all computing nodes which experience a node failure within a predefined time period (e.g., 10 minutes) regardless of their physical location. Therefore, it is required to regularly update the set of nodes in this vicinity and also unlike the others, it can be used only for offline behavior analysis after node failure. However, the results of offline behavior analysis can be used in online anomaly detection.

Figure 9 illustrates job failures on Taurus. Each red circle represents a failure occurred on a node on a specific date. The horizontal concentration of red circles indicates several failures on a node, while the vertical concentration of circles represents simultaneous failures on several nodes. In several cases, a temporal correlation among node failures is observable. However, in most cases, the temporal correlation is supported by an even stronger spatial correlation (horizontal concentration of points in Figure 3 and Figure 9).

Application of the proposed anomaly detection method on time of failure vicinity provided different results according to the cause of failures. When failures were caused by misbehaving global shared resources, such as the distributed file system, the results were accurate and reliable. However, for other causes of failures, such as misuse of local resources or sudden network interruptions, a high number of false positives hindered the successful anomaly detection.

III-E Results and Discussions

Subsection III-E summarizes the outcome of Subsections III-A, III-B, III-C, and III-D in which the efficiency of anomaly detection was studied in various vicinities. According to these preliminary results, the impact of failure detection via the proposed method inside resource allocation and time of failure vicinities of Taurus is negligible and thus, should be avoided. Failure detection inside the Physical location vicinity, on the other hand, has a high impact on the accuracy of the final results. It also became evident that the application of the proposed method outside the hardware architecture vicinity of nodes can significantly degrade the accuracy of the failure detection mechanism.