P*: A Model of Pilot-Abstractions

TL;DR

This paper introduces a comprehensive, unifying model for Pilot-Jobs called P*, aiming to improve interoperability, extensibility, and understanding across various distributed computing frameworks.

Contribution

It provides a minimal yet complete model of Pilot-Jobs, maps existing frameworks to it, and develops an interoperable API validated across multiple infrastructures.

Findings

The P* model successfully maps to frameworks like Condor and DIANE.

The Pilot-API enables interoperability among different Pilot-Job frameworks.

The model is extended to Pilot-Data applications.

Abstract

Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jobs which can be used to define, compare and contrast different implementations. Often Pilot-Job implementations are strongly coupled to the distributed cyber-infrastructure they were originally designed for. These factors present a barrier to extensibility and interoperability. This pa- per is an attempt to (i) provide a minimal but complete model (P*) of Pilot-Jobs, (ii) establish the generality of the P* Model by mapping various existing and well known Pilot-Job frameworks such as Condor and DIANE to P*, (iii) derive an interoperable and extensible API for the P* Model (Pilot-API), (iv) validate the implementation of the Pilot-API by concurrently using multiple distinct Pilot-Job frameworks on distinct production distributed cyberinfrastructures, and (v) apply the P* Model to Pilot-Data.