Nextpie: a web-based reporting tool and database for reproducible nextflow pipelines

TL;DR

Nextpie is a web-based tool that helps manage and visualize resource usage data from Nextflow pipelines, making genomic data analysis more efficient and reproducible.

Contribution

Nextpie introduces a database and reporting system for Nextflow pipelines, enabling streamlined resource usage tracking and visualization.

Findings

Nextpie stores pipeline resource usage in a relational database for easier analysis.

The tool provides interactive visualizations and a reporting interface for pipeline performance.

Nextpie is publicly available with example data and documentation.

Abstract

High-throughput genomic data analysis consists of the inexorably intertwined inputs and outputs of a vast array of bioinformatic analysis tools. To guarantee streamlined and reproducible analyses, the often complex data analysis pipelines need to be run using workflow management tools. Nextflow is one popular tool commonly used to automate such pipelines. Nextflow records key pipeline data, such as the submission time, start time, completion time, CPU usage, memory usage, and disk usage for each task run. These data are stored in log files, often scattered across a file system. Therefore, aggregating information about resource usage critical for the optimization of Nextflow pipelines and improving reproducibility, as well as parsing and managing such log data, can quickly become cumbersome.

Here, we present a web-based tool, Nextpie, which provides both a database and a reporting tool for Nextflow pipelines. Nextpie stores comprehensive resource usage information in a relational database, thus facilitating and accelerating the performance of a variety of data analyses and interactive visualizations, providing an easily comprehensible overview of a pipeline’s resource usage.

The Nextpie source code, user documentation, an SQLite database with test data, and a Nextflow example pipeline are available at GitHub (https://github.com/bishwaG/Nextpie).

Figures