Essential guidelines for computational method benchmarking

TL;DR

This paper provides essential guidelines for designing and conducting high-quality benchmarking studies in computational biology, ensuring accurate, unbiased, and informative comparisons of different computational methods.

Contribution

It offers practical recommendations and best practices for benchmarking studies, based on extensive experience in computational biology.

Findings

Emphasizes the importance of careful benchmarking design

Highlights common pitfalls and biases to avoid

Provides a framework for transparent and reproducible benchmarking

Abstract

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology.