A Framework for a Multiagent-based Scheduling of Parallel Jobs

TL;DR

This paper introduces a multiagent framework for scheduling parallel jobs that reduces communication bottlenecks and improves system performance by distributing communication latency among processors.

Contribution

It presents a novel multiagent approach for parallel job scheduling that outperforms traditional master-slave methods, especially at higher processor counts.

Findings

Multiagent scheduling reduces communication bottlenecks.

Performance improves with more processors.

Lower average parallel cost in simulations.

Abstract

This paper presents a multiagent approach as a paradigm for scheduling parallel jobs in a parallel system. Scheduling parallel jobs is performed as a means to balance the load of a system in order to improve the performance of a parallel application. Parallel job scheduling is presented as a mapping between two graphs: one represents the dependency of jobs and the other represents the interconnection among processors. The usual implementation of parallel job scheduling algorithms is via the master-slave paradigm. The master-slave paradigm has inherent communication bottleneck that reduces the performance of the system when more processors are needed to process the jobs. The multiagent approach attempts to distribute the communication latency among the processors which improves the performance of the system as the number of participating processors increases. Presented in this paper is a framework for the behavior of an autonomous agent that cooperates with other agents to achieve a community goal of minimizing the processing time. Achieving this goal means an agent must truthfully share information with other agents via {\em normalization}, {\em task sharing}, and {\em result sharing} procedures. The agents consider a parallel scientific application as a finite-horizon game where truthful information sharing results into performance improvement for the parallel application. The performance of the multiagent-based algorithm is compared to that of an existing one via a simulation of the wavepacket dynamics using the quantum trajectory method (QTM) as a test application. The average parallel cost of running the QTM using the multiagent-based system is lower at higher number of processors.