Managing Cloud networking costs for data-intensive applications by provisioning dedicated network links
Igor Sfiligoi, Michael Hare, David Schultz, Frank W\"urthwein,, Benedikt Riedel, Tom Hutton, Steve Barnet, Vladimir Brik

TL;DR
This paper discusses how dedicated network links can significantly reduce data transfer costs for data-intensive scientific applications in the cloud, demonstrated through a high-performance IceCube compute setup across major cloud providers.
Contribution
It presents a detailed case study of provisioning dedicated network links across AWS, Azure, and GCP to support high-throughput scientific computing, including cost analysis and technical insights.
Findings
Dedicated links reduced egress costs for data-intensive workloads.
Provisioning multiple cloud providers enabled approximately 100 Gbps egress capacity.
The setup demonstrated technical feasibility and highlighted cost benefits and limitations.
Abstract
Many scientific high-throughput applications can benefit from the elastic nature of Cloud resources, especially when there is a need to reduce time to completion. Cost considerations are usually a major issue in such endeavors, with networking often a major component; for data-intensive applications, egress networking costs can exceed the compute costs. Dedicated network links provide a way to lower the networking costs, but they do add complexity. In this paper we provide a description of a 100 fp32 PFLOPS Cloud burst in support of IceCube production compute, that used Internet2 Cloud Connect service to provision several logically-dedicated network links from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and Google Cloud Platform, that in aggregate enabled approximately 100 Gbps egress capability to on-prem storage. It provides technical details about the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
