Assessment of Multiple-Biomarker Classifiers: fundamental principles and a proposed strategy

TL;DR

This paper reviews the assessment of multiple-biomarker classifiers, emphasizing the importance of accounting for finite training sample uncertainty, and proposes a three-level strategy to improve classifier evaluation and stability.

Contribution

It introduces a novel three-level strategy for classifier assessment that incorporates uncertainty estimation from training samples to enhance reliability.

Findings

Finite training sample uncertainty significantly impacts classifier stability.

A three-level assessment strategy improves classifier evaluation.

Resources for implementing the approach are reviewed.

Abstract

The multiple-biomarker classifier problem and its assessment are reviewed against the background of some fundamental principles from the field of statistical pattern recognition, machine learning, or the recently so-called "data science". A narrow reading of that literature has led many authors to neglect the contribution to the total uncertainty of performance assessment from the finite training sample. Yet the latter is a fundamental indicator of the stability of a classifier; thus its neglect may be contributing to the problematic status of many studies. A three-level strategy is proposed for moving forward in this field. The lowest level is that of construction, where candidate features are selected and the choice of classifier architecture is made. At that point, the effective dimensionality of the classifier is estimated and used to size the next level of analysis, a pilot study on previously unseen cases. The total (training and testing) uncertainty resulting from the pilot study is, in turn, used to size the highest level of analysis, a pivotal study with a target level of uncertainty. Some resources available in the literature for implementing this approach are reviewed. Although the concepts explained in the present article may be fundamental and straightforward for many researchers in the machine learning community they are subtle for many practitioners, for whom we provided a general advice for the best practice in \cite{Shi2010MAQCII} and elaborate here in the present paper.