Demand-driven provisioning of Kubernetes-like resources in OSG
Igor Sfiligoi, Frank W\"urthwein, Jeff Dost, Brian Lin, David, Schultz

TL;DR
This paper presents a demand-driven provisioning system for Kubernetes-like resources in the OSG, enabling efficient resource utilization across multiple providers including Lancium, through a simple and automated mechanism.
Contribution
It introduces a novel demand-driven provisioner for Kubernetes resources that extends existing backfill containers, supporting multiple providers with minimal complexity.
Findings
Successfully provisioned resources from NRP and Lancium using the new system
Automated matchmaking improves resource utilization efficiency
The simple logic allows easy extension and low-cost deployment
Abstract
The OSG-operated Open Science Pool is an HTCondor-based virtual cluster that aggregates resources from compute clusters provided by several organizations. Most of the resources are not owned by OSG, so demand-based dynamic provisioning is important for maximizing usage without incurring excessive waste. OSG has long relied on GlideinWMS for most of its resource provisioning needs but is limited to resources that provide a Grid-compliant Compute Entrypoint. To work around this limitation, the OSG Software Team has developed a glidein container that resource providers could use to directly contribute to the OSPool. The problem of that approach is that it is not demand-driven, relegating it to backfill scenarios only. To address this limitation, a demand-driven direct provisioner of Kubernetes resources has been developed and successfully used on the NRP. The setup still relies on the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDistributed and Parallel Computing Systems · Scientific Computing and Data Management · Cloud Computing and Resource Management
