Managing Uncertainty: A Case for Probabilistic Grid Scheduling

TL;DR

This paper proposes a probabilistic scheduling approach for heterogeneous Grid environments, utilizing historical job data to improve deadline and economy-based scheduling in a complex, multi-application setting.

Contribution

It introduces a novel probabilistic scheduling framework that leverages statistical properties of past job executions for better resource management in Grid computing.

Findings

Analyzed six months of Grid application data for insights.

Demonstrated the effectiveness of probabilistic scheduling models.

Discussed management approaches suitable for autonomous Grid scheduling.

Abstract

The Grid technology is evolving into a global, service-orientated architecture, a universal platform for delivering future high demand computational services. Strong adoption of the Grid and the utility computing concept is leading to an increasing number of Grid installations running a wide range of applications of different size and complexity. In this paper we address the problem of elivering deadline/economy based scheduling in a heterogeneous application environment using statistical properties of job historical executions and its associated meta-data. This approach is motivated by a study of six-month computational load generated by Grid applications in a multi-purpose Grid cluster serving a community of twenty e-Science projects. The observed job statistics, resource utilisation and user behaviour is discussed in the context of management approaches and models most suitable for supporting a probabilistic and autonomous scheduling architecture.