Benchmarking Different Application Types across Heterogeneous Cloud Compute Services

TL;DR

This paper benchmarks the performance of diverse application types across heterogeneous cloud compute services, providing insights into cost, latency, and energy efficiency to guide optimal cloud resource utilization.

Contribution

It introduces curated benchmark datasets for three application types across different domains, aiding researchers in evaluating cloud compute service performance.

Findings

Benchmark datasets for DNN inference, ML inference, and video transcoding.

Performance insights across heterogeneous cloud servers.

Guidelines for cost and energy-efficient cloud deployment.

Abstract

Infrastructure as a Service (IaaS) clouds have become the predominant underlying infrastructure for the operation of modern and smart technology. IaaS clouds have proven to be useful for multiple reasons such as reduced costs, increased speed and efficiency, and better reliability and scalability. Compute services offered by such clouds are heterogeneous -- they offer a set of architecturally diverse machines that fit efficiently executing different workloads. However, there has been little study to shed light on the performance of popular application types on these heterogeneous compute servers across different clouds. Such a study can help organizations to optimally (in terms of cost, latency, throughput, consumed energy, carbon footprint, etc.) employ cloud compute services. At HPCC lab, we have focused on such benchmarks in different research projects and, in this report, we curate those benchmarks in a single document to help other researchers in the community using them. Specifically, we introduce our benchmarks datasets for three application types in three different domains, namely: Deep Neural Networks (DNN) Inference for industrial applications, Machine Learning (ML) Inference for assistive technology applications, and video transcoding for multimedia use cases.