On unbiased performance evaluation for protein inference

TL;DR

This paper discusses the importance of unbiased performance evaluation in protein inference algorithms, arguing against grid search parameter tuning to prevent overestimation of algorithm accuracy.

Contribution

It clarifies why grid search should not be used for parameter tuning in performance evaluation, promoting fair comparison of protein inference methods.

Findings

Grid search can lead to over-estimated performance metrics.

Unbiased evaluation requires fixed parameters, not tuned on test data.

The paper emphasizes fair comparison practices in bioinformatics algorithms.

Abstract

This letter is a response to the comments of Serang (2012) on Huang and He (2012) in Bioinformatics. Serang (2012) claimed that the parameters for the Fido algorithm should be specified using the grid search method in Serang et al. (2010) so as to generate a deserved accuracy in performance comparison. It seems that it is an argument on parameter tuning. However, it is indeed the issue of how to conduct an unbiased performance evaluation for comparing different protein inference algorithms. In this letter, we would explain why we don't use the grid search for parameter selection in Huang and He (2012) and show that this procedure may result in an over-estimated performance that is unfair to competing algorithms. In fact, this issue has also been pointed out by Li and Radivojac (2012).