An SLA-based Advisor for Placement of HPC Jobs on Hybrid Clouds

TL;DR

This paper presents an SLA-based advisory system that helps HPC users decide whether to run jobs on local clusters or cloud resources, optimizing for cost, performance, and availability in hybrid cloud environments.

Contribution

It introduces a novel advisory service for HPC job placement on hybrid clouds, considering trade-offs and providing practical guidance beyond performance comparison.

Findings

The advisor effectively reduces costs and turnaround times.

Evaluation with seismic processing shows practical benefits.

The approach is adaptable to various HPC applications.

Abstract

Several scientific and industry applications require High Performance Computing (HPC) resources to process and/or simulate complex models. Not long ago, companies, research institutes, and universities used to acquire and maintain on-premise computer clusters; but, recently, cloud computing has emerged as an alternative for a subset of HPC applications. This poses a challenge to end-users, who have to decide where to run their jobs: on local clusters or burst to a remote cloud service provider. While current research on HPC cloud has focused on comparing performance of on-premise clusters against cloud resources, we build on top of existing efforts and introduce an advisory service to help users make this decision considering the trade-offs of resource costs, performance, and availability on hybrid clouds. We evaluated our service using a real test-bed with a seismic processing application based on Full Waveform Inversion; a technique used by geophysicists in the oil & gas industry and earthquake prediction. We also discuss how the advisor can be used for other applications and highlight the main lessons learned constructing this service to reduce costs and turnaround times.