Building a continuous benchmarking ecosystem in bioinformatics

TL;DR

This paper outlines a vision for a unified platform to streamline and standardize benchmarking in bioinformatics.

Contribution

It proposes a wish list for a computational platform to orchestrate reproducible and standardized benchmark studies.

Findings

A computational platform could unify benchmarking by managing software environments and workflows.

Standardized definitions and interfaces are needed for reproducible benchmarking.

The paper highlights the importance of integrating benchmarking with computing infrastructure.

Abstract

Benchmarking, which involves collecting reference datasets and demonstrating method performance, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.

Figures