Deployment of Elastic Virtual Hybrid Clusters Across Cloud Sites

TL;DR

This paper presents an architecture for deploying elastic virtual clusters across multiple cloud sites, enabling scalable and geographically distributed high-performance and high-throughput computing.

Contribution

It introduces an open-source, IaC-based architecture for multi-cloud virtual elastic clusters with automated deployment and secure communication across sites.

Findings

Hybrid clusters improve resource utilization for parallel jobs

Automated deployment simplifies multi-cloud cluster setup

Secure VPN tunneling enables seamless communication

Abstract

Virtual clusters are widely used computing platforms than can be deployed in multiple cloud platforms. The ability to dynamically grow and shrink the number of nodes has paved the way for customised elastic computing both for High Performance Computing and High Throughput Computing workloads. However, elasticity is typically restricted to a single cloud site, thus hindering the ability to provision computational resources from multiple geographically distributed cloud sites. To this aim, this paper introduces an architecture of open-source components that coherently deploy a virtual elastic cluster across multiple cloud sites to perform large-scale computing. These hybrid virtual elastic clusters are automatically deployed and configured using an Infrastructure as Code (IaC) approach on a distributed hybrid testbed that spans different organizations, including on-premises and public clouds, supporting automated tunneling of communications across the cluster nodes with advanced VPN topologies. The results indicate that cluster-based computing of embarrassingly parallel jobs can benefit from hybrid virtual clusters that aggregate computing resources from multiple cloud back-ends and bring them together into a dedicated, albeit virtual network. The work presented in this article has been partially funded by the European Union's (EU) Horizon 2020 research project DEEP Hybrid-DataCloud (grant agreement No 777435).