Intelligently-automated facilities expansion with the HEPCloud Decision Engine

TL;DR

This paper presents the HEPCloud Decision Engine, an intelligent system that automates resource provisioning across multiple cloud and HPC platforms to support large-scale physics experiments efficiently.

Contribution

It introduces a modular IDSS that automates resource provisioning across diverse platforms, considering organizational rules and budget constraints.

Findings

Successfully integrates multiple cloud and HPC resources

Supports elastic expansion for high data volume demands

Automates decision-making for resource allocation

Abstract

The next generation of High Energy Physics experiments are expected to generate exabytes of data---two orders of magnitude greater than the current generation. In order to reliably meet peak demands, facilities must either plan to provision enough resources to cover the forecasted need, or find ways to elastically expand their computational capabilities. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and to choose an appropriate scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources---spanning multiple cloud providers, multiple HPC centers, and grid computing federations.