RNF: a general framework to evaluate NGS read mappers

TL;DR

This paper introduces RNF, a universal format and software framework for evaluating NGS read mappers by encoding read origin information, enabling standardized and flexible assessment of mapping accuracy and parameter effects.

Contribution

The authors developed RNF, a generic read naming format, and an associated software package that simplifies and standardizes the evaluation of NGS read mappers across different simulators.

Findings

RNF enables consistent evaluation of read mappers.

The framework supports analysis of mapping qualities and contamination effects.

It facilitates comparison across different simulation tools.

Abstract

Aligning reads to a reference sequence is a fundamental step in numerous bioinformatics pipelines. As a consequence, the sensitivity and precision of the mapping tool, applied with certain parameters to certain data, can critically affect the accuracy of produced results (e.g., in variant calling applications). Therefore, there has been an increasing demand of methods for comparing mappers and for measuring effects of their parameters. Read simulators combined with alignment evaluation tools provide the most straightforward way to evaluate and compare mappers. Simulation of reads is accompanied by information about their positions in the source genome. This information is then used to evaluate alignments produced by the mapper. Finally, reports containing statistics of successful read alignments are created. In default of standards for encoding read origins, every evaluation tool has to be made explicitly compatible with the simulator used to generate reads. In order to solve this obstacle, we have created a generic format RNF (Read Naming Format) for assigning read names with encoded information about original positions. Futhermore, we have developed an associated software package RNF containing two principal components. MIShmash applies one of popular read simulating tools (among DwgSim, Art, Mason, CuReSim etc.) and transforms the generated reads into RNF format. LAVEnder evaluates then a given read mapper using simulated reads in RNF format. A special attention is payed to mapping qualities that serve for parametrization of ROC curves, and to evaluation of the effect of read sample contamination.