Bulk Scheduling with the DIANA Scheduler

TL;DR

The paper presents the DIANA Scheduler, an adaptive, performance-aware, and economy-guided meta-scheduler designed for data-intensive sciences, optimizing resource management across multiple locations considering network costs.

Contribution

It introduces a novel adaptive scheduling algorithm that accounts for data location, network performance, and compute capabilities for efficient global resource management.

Findings

Significant performance improvements with DIANA scheduling.

Effective handling of bulk scheduling tasks.

Enhanced resource utilization and efficiency.

Abstract

Results from the research and development of a Data Intensive and Network Aware (DIANA) scheduling engine, to be used primarily for data intensive sciences such as physics analysis, are described. In Grid analyses, tasks can involve thousands of computing, data handling, and network resources. The central problem in the scheduling of these resources is the coordinated management of computation and data at multiple locations and not just data replication or movement. However, this can prove to be a rather costly operation and efficient sing can be a challenge if compute and data resources are mapped without considering network costs. We have implemented an adaptive algorithm within the so-called DIANA Scheduler which takes into account data location and size, network performance and computation capability in order to enable efficient global scheduling. DIANA is a performance-aware and economy-guided Meta Scheduler. It iteratively allocates each job to the site that is most likely to produce the best performance as well as optimizing the global queue for any remaining jobs. Therefore it is equally suitable whether a single job is being submitted or bulk scheduling is being performed. Results indicate that considerable performance improvements can be gained by adopting the DIANA scheduling approach.