Integrating Abstractions to Enhance the Execution of Distributed Applications

TL;DR

This paper presents a middleware that integrates abstractions for distributed applications, resources, and execution to improve scalability, performance, and analysis capabilities in distributed computing environments.

Contribution

It introduces a pilot-based middleware that unifies application and resource abstractions, enabling flexible execution and detailed performance analysis of distributed applications.

Findings

Using multiple resources improves performance.

Late-binding scheduling enhances flexibility.

Backfill scheduling optimizes resource utilization.

Abstract

One of the factors that limits the scale, performance, and sophistication of distributed applications is the difficulty of concurrently executing them on multiple distributed computing resources. In part, this is due to a poor understanding of the general properties and performance of the coupling between applications and dynamic resources. This paper addresses this issue by integrating abstractions representing distributed applications, resources, and execution processes into a pilot-based middleware. The middleware provides a platform that can specify distributed applications, execute them on multiple resource and for different configurations, and is instrumented to support investigative analysis. We analyzed the execution of distributed applications using experiments that measure the benefits of using multiple resources, the late-binding of scheduling decisions, and the use of backfill scheduling.