MultiCloud Resource Management using Apache Mesos for Planned Integration with Apache Airavata

TL;DR

This paper explores integrating Apache Mesos with Apache Airavata to enable multi-cloud resource management and scheduling, including handling VMs without public IPs, to improve job distribution across clouds.

Contribution

It proposes a novel meta-scheduler design using Apache Mesos for multi-cloud environments, addressing resource visibility and job scheduling challenges without requiring public IPs.

Findings

Initial results demonstrate resource management across multiple clouds.

The approach enables scheduling of jobs on VMs without public IPs.

Next steps include developing and testing the meta-scheduler.

Abstract

We discuss initial results and our planned approach for incorporating Apache Mesos based resource management that will enable design and development of scheduling strategies for Apache Airavata jobs so that they can be launched on multiple clouds, wherein several VMs do not have Public IP addresses. We present initial work and next steps on the design of a meta-scheduler using Apache Mesos. Apache Mesos presents a unified view of resources available across several clouds and clusters. Our meta-scheduler can potentially examine and identify the cases where multiple small jobs have been submitted by the same scientists and then redirect job from the same community account or user to different clusters. Our approach uses a NAT firewall to make nodes/VMs, without a Public IP, visible to Mesos for the unified view.