# Building a continuous benchmarking ecosystem in bioinformatics

## TL;DR

This paper outlines a vision for a unified platform to streamline and standardize benchmarking in bioinformatics.

## Contribution

It proposes a wish list for a computational platform to orchestrate reproducible and standardized benchmark studies.

## Key findings

- A computational platform could unify benchmarking by managing software environments and workflows.
- Standardized definitions and interfaces are needed for reproducible benchmarking.
- The paper highlights the importance of integrating benchmarking with computing infrastructure.

## Abstract

Benchmarking, which involves collecting reference datasets and demonstrating method performance, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark studies, this Perspective sheds light on a wish list for a computational platform to orchestrate benchmark studies. We discuss various ideas for organizing reproducible software environments, formally defining benchmarks, orchestrating standardized workflows, and how they interface with computing infrastructure.