Machine Learning Algorithms for Active Monitoring of High Performance Computing as a Service (HPCaaS) Cloud Environments

TL;DR

This paper investigates the use of machine learning to identify engineering applications running on cloud-based HPC environments, utilizing privacy-preserving billing data to classify different computational workloads.

Contribution

It introduces a method for classifying HPC applications on cloud infrastructure using privacy-preserving billing features and demonstrates its effectiveness with real-world engineering codes.

Findings

Successful classification of different HPC applications using billing data

Demonstrated privacy-preserving application identification in cloud HPC environments

Applicable to various cloud providers and HPC workloads

Abstract

Cloud computing provides ubiquitous and on-demand access to vast reconfigurable resources that can meet any computational need. Many service models are available, but the Infrastructure as a Service (IaaS) model is particularly suited to operate as a high performance computing (HPC) platform, by networking large numbers of cloud computing nodes. We used the Pacific Northwest National Laboratory (PNNL) cloud computing environment to perform our experiments. A number of cloud computing providers such as Amazon Web Services, Microsoft Azure, or IBM Cloud, offer flexible and scalable computing resources. This paper explores the viability identifying types of engineering applications running on a cloud infrastructure configured as an HPC platform using privacy preserving features as input to statistical models. The engineering applications considered in this work include MCNP6, a radiation transport code developed by Los Alamos National Laboratory, OpenFOAM, an open source computational fluid dynamics code, and CADO-NFS, a numerical implementation of the general number field sieve algorithm used for prime number factorization. Our experiments use the OpenStack cloud management tool to create a cloud HPC environment and the privacy preserving Ceilometer billing meters as classification features to demonstrate identification of these applications.