Spatiotemporal Modeling of Node Temperatures in Supercomputers

TL;DR

This paper develops a statistical spatiotemporal model using Gaussian processes and GMRFs to analyze and optimize node temperature management in supercomputing clusters, aiming to reduce cooling costs and prevent overheating.

Contribution

It introduces a novel combination of distribution modeling and Gaussian Markov random fields for detailed temperature analysis in supercomputers.

Findings

Identified causes of overheating episodes.

Provided a framework for predicting temperature trends.

Enabled assessment of cooling efficiency improvements.

Abstract

Los Alamos National Laboratory (LANL) is home to many large supercomputing clusters. These clusters require an enormous amount of power (~500-2000 kW each), and most of this energy is converted into heat. Thus, cooling the components of the supercomputer becomes a critical and expensive endeavor. Recently a project was initiated to optimize the cooling system used to cool one of the rooms housing three of these large clusters and develop a general good-practice procedure for reducing cooling costs and monitoring other machine rooms. This work focuses on the statistical approach used to quantify the effect that several cooling changes to the room had on the temperatures of the individual nodes of the computers. The largest cluster in the room has 1600 nodes that run a variety of jobs during general use. Since extremes temperatures are important, a Normal distribution plus generalized Pareto distribution for the upper tail is used to model the marginal distribution, along with a Gaussian process copula to account for spatio-temporal dependence. A Gaussian Markov random field (GMRF) model is used to model the spatial and/or temporal effects on the node temperatures as the cooling changes take place. This model is then used to assess the condition of the node temperatures after each change to the room. The analysis approach was used to uncover the cause of a problematic episode of overheating nodes on one of the supercomputing clusters. The next step is to also use the model to estimate the trend in node temperatures due to an increase in supply air temperature and ultimately decide when any further temperature increases would become unsafe. This same process can be applied to reduce the cooling expenses for other data centers as well.