dynamicMF: A Matrix Factorization Approach to Monitor Resource Usage in High Performance Computing Systems

TL;DR

This paper introduces dynamicMF, a tensor-based matrix factorization technique that reduces high-dimensional HPC resource data to low-dimensional signals for real-time system monitoring and anomaly detection.

Contribution

The paper presents a novel dynamic matrix factorization method for low-dimensional representation of HPC resource data, enabling efficient anomaly detection.

Findings

Identified anomalies correlate with actual system events.

Effective reduction of multi-dimensional data to low-dimensional signals.

Improved real-time monitoring capabilities for HPC systems.

Abstract

High performance computing (HPC) facilities consist of a large number of interconnected computing units (or nodes) that execute highly complex scientific simulations to support scientific research. Monitoring such facilities, in real-time, is essential to ensure that the system operates at peak efficiency. Such systems are typically monitored using a variety of measurement and log data which capture the state of the various components within the system at regular intervals of time. As modern HPC systems grow in capacity and complexity, the data produced by current resource monitoring tools is at a scale that it is no longer feasible to be visually monitored by analysts. We propose a method that transforms the multi-dimensional output of resource monitoring tools to a low dimensional representation that facilitates the understanding of the behavior of a High Performance Computing (HPC) system. The proposed method automatically extracts the low-dimensional signal in the data which can be used to track the system efficiency and identify performance anomalies. The method models the resource usage data as a three dimensional tensor (capturing resource usage of all compute nodes for difference resources over time). A dynamic matrix factorization algorithm, called dynamicMF, is proposed to extract a low-dimensional temporal signal for each node, which is subsequently fed into an anomaly detector. Results on resource usage data collected from the Lonestar 4 system at the Texas Advanced Computing Center show that the identified anomalies are correlated with actual anomalous events reported in the system log messages.