JMS: A workflow management system and web-based cluster front-end for the Torque resource manager

TL;DR

JMS is an open-source workflow management system that simplifies creating and executing complex computational pipelines on distributed clusters, integrating with Torque and supporting web-based interfaces for scientific research.

Contribution

The paper introduces JMS, a novel web-based workflow management system that seamlessly integrates with Torque for managing complex scientific pipelines on clusters.

Findings

JMS enables easy creation of complex workflows.

It integrates with Torque for efficient job management.

Used successfully in bioinformatics pipelines.

Abstract

Motivation: Complex computational pipelines are becoming a staple of modern scientific research. Often these pipelines are resource intensive and require days of computing time. In such cases, it makes sense to run them over distributed computer clusters where they can take advantage of the aggregated resources of many powerful computers. In addition to this, researchers often want to integrate their workflows into their own web servers. In these cases, software is needed to manage the submission of jobs from the web interface to the cluster and then return the results once the job has finished executing. Results: We have developed the Job Management System (JMS), a workflow management system and interface for the Torque resource manager. The JMS provides users with a user-friendly interface for creating complex workflows with multiple stages. It integrates this workflow functionality with Torque, a tool that is used to control and manage batch jobs on distributed computing clusters. The JMS can be used by researchers to build and run complex computational pipelines and provides functionality to include these pipelines in external interfaces. The JMS is currently being used to house a number of structural bioinformatics pipelines at the Research Unit in Bioinformatics (RUBi) at Rhodes University. Availability: The JMS is an open-source project and is freely available at https://github.com/RUBi-ZA/JMS