MirLibSpark: A Scalable NGS Plant MicroRNA Prediction Pipeline for Multi-Library Functional Annotation

TL;DR

MirLibSpark is a scalable, fast, and accurate distributed pipeline for plant microRNA prediction and functional annotation from large transcriptomic datasets, leveraging Apache Spark for high-volume processing.

Contribution

It introduces the first distributed, fully automated plant miRNA predictor that improves speed and accuracy for large-scale data analysis using Spark.

Findings

Highest processing speed among current methods

Improved prediction accuracy over standard tools

Fully automated pipeline for large datasets

Abstract

The emergence of the Next Generation Sequencing increases drastically the volume of transcriptomic data. Although many standalone algorithms and workflows for novel microRNA (miRNA) prediction have been proposed, few are designed for processing large volume of sequence data from large genomes, and even fewer further annotate functional miRNAs by analyzing multiple libraries. We propose an improved pipeline for a high volume data facility by implementing mirLibSpark based on the Apache Spark framework. This pipeline is the fastest actual method, and provides an accuracy improvement compared to the standard. In this paper, we deliver the first distributed functional miRNA predictor as a standalone and fully automated package. It is an efficient and accurate miRNA predictor with functional insight. Furthermore, it compiles with the gold-standard requirement on plant miRNA predictions.