LIDC: A Location Independent Multi-Cluster Computing Framework for Data Intensive Science

TL;DR

This paper presents LIDC, a decentralized framework enabling location-independent and dynamic placement of data-intensive computations across geographically distributed clusters using semantic naming, overcoming limitations of centralized control systems.

Contribution

Introduces a decentralized control plane for multi-cluster computing that uses semantic names for flexible, location-independent task placement in data-intensive science workflows.

Findings

Enables location-independent job placement across clusters.

Supports dynamic compute placement without prior infrastructure knowledge.

Improves flexibility over traditional centralized systems.

Abstract

Scientific communities are increasingly using geographically distributed computing platforms. The current methods of compute placement predominantly use logically centralized controllers such as Kubernetes (K8s) to match tasks to available resources. However, this centralized approach is unsuitable in multi-organizational collaborations. Furthermore, workflows often need to use manual configurations tailored for a single platform and cannot adapt to dynamic changes across infrastructure. Our work introduces a decentralized control plane for placing computations on geographically dispersed compute clusters using semantic names. We assign semantic names to computations to match requests with named Kubernetes (K8s) service endpoints. We show that this approach provides multiple benefits. First, it allows placement of computational jobs to be independent of location, enabling any cluster with sufficient resources to execute the computation. Second, it facilitates dynamic compute placement without requiring prior knowledge of cluster locations or predefined configurations.