Comprehensive Plugin-Based Monitoring of Nexflow Workflow Executions

TL;DR

This paper introduces a comprehensive, plugin-based monitoring system for Nextflow workflows that enhances flexibility and detail without requiring custom Nextflow modifications.

Contribution

It presents a new monitoring plugin compatible with wf-instances, enabling detailed, online monitoring of Nextflow workflows without custom forks or recompilation.

Findings

Provides more detailed monitoring than built-in Nextflow tools.

Allows online monitoring during workflow execution.

Eliminates need for custom Nextflow forks or recompilation.

Abstract

Nextflow is a workflow management system commonly used in fields like bioinformatics and earth observation. It coordinates distributed data processing of various tools as an acyclic sequence of tasks while using, containerization (e.g., Docker), orchestration (e.g., Kubernetes), or batch processing (e.g., SLURM). Monitoring such workflow executions can be challenging but aids performance analysis, debugging, and data provenance. Besides Nexflow's basic built-in monitoring, the wf-commons tool for creating wf-instances is widely regarded as the standard in the Nextflow community. The monitoring plugin we develpoed provides a more detailed and flexible alternative compatible with wf-instances while removing the need for a custom Nextflow fork by using Nextflow's plug-in mechanism (version 21.10), optional direct .jar file changes of static artifacts without recompilation and allows online monitoring during execution.