Managing Cloud networking costs for data-intensive applications by provisioning dedicated network links

TL;DR

This paper discusses how dedicated network links can significantly reduce data transfer costs for data-intensive scientific applications in the cloud, demonstrated through a high-performance IceCube compute setup across major cloud providers.

Contribution

It presents a detailed case study of provisioning dedicated network links across AWS, Azure, and GCP to support high-throughput scientific computing, including cost analysis and technical insights.

Findings

Dedicated links reduced egress costs for data-intensive workloads.

Provisioning multiple cloud providers enabled approximately 100 Gbps egress capacity.

The setup demonstrated technical feasibility and highlighted cost benefits and limitations.

Abstract

Many scientific high-throughput applications can benefit from the elastic nature of Cloud resources, especially when there is a need to reduce time to completion. Cost considerations are usually a major issue in such endeavors, with networking often a major component; for data-intensive applications, egress networking costs can exceed the compute costs. Dedicated network links provide a way to lower the networking costs, but they do add complexity. In this paper we provide a description of a 100 fp32 PFLOPS Cloud burst in support of IceCube production compute, that used Internet2 Cloud Connect service to provision several logically-dedicated network links from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and Google Cloud Platform, that in aggregate enabled approximately 100 Gbps egress capability to on-prem storage. It provides technical details about the provisioning process, the benefits and limitations of such a setup and an analysis of the costs incurred.