Towards a Dynamic Composability Approach for using Heterogeneous Systems in Remote Sensing

TL;DR

This paper introduces a novel dynamic composability approach for heterogeneous systems in remote sensing, integrating supercomputing and cloud resources to enhance scientific workflows involving AI and data-driven applications.

Contribution

It presents a new architecture for federating supercomputing and cloud systems, enabling flexible, re-programmable, and efficient scientific computing environments.

Findings

Successful integration of Expanse supercomputer with Nautilus Kubernetes cluster

Demonstration of workflow bridging edge sensing, AI, and physics simulation

Enhanced scientific workflow flexibility and resource utilization

Abstract

Influenced by the advances in data and computing, the scientific practice increasingly involves machine learning and artificial intelligence driven methods which requires specialized capabilities at the system-, science- and service-level in addition to the conventional large-capacity supercomputing approaches. The latest distributed architectures built around the composability of data-centric applications led to the emergence of a new ecosystem for container coordination and integration. However, there is still a divide between the application development pipelines of existing supercomputing environments, and these new dynamic environments that disaggregate fluid resource pools through accessible, portable and re-programmable interfaces. New approaches for dynamic composability of heterogeneous systems are needed to further advance the data-driven scientific practice for the purpose of more efficient computing and usable tools for specific scientific domains. In this paper, we present a novel approach for using composable systems in the intersection between scientific computing, artificial intelligence (AI), and remote sensing domain. We describe the architecture of a first working example of a composable infrastructure that federates Expanse, an NSF-funded supercomputer, with Nautilus, a Kubernetes-based GPU geo-distributed cluster. We also summarize a case study in wildfire modeling, that demonstrates the application of this new infrastructure in scientific workflows: a composed system that bridges the insights from edge sensing, AI and computing capabilities with a physics-driven simulation.