Effects of Training Data Quality on Classifier Performance

TL;DR

This study systematically investigates how the quality of training data impacts the performance of various classifiers in metagenomic analysis, revealing degradation patterns and increased classifier congruence with data quality decline.

Contribution

It provides a comprehensive experimental analysis of classifier robustness to training data degradation across multiple models in a biological context.

Findings

Classifier performance deteriorates with data quality degradation.

All classifiers show similar breakdown behavior under poor data conditions.

Classifier decisions become more homogeneous as data quality worsens.

Abstract

We describe extensive numerical experiments assessing and quantifying how classifier performance depends on the quality of the training data, a frequently neglected component of the analysis of classifiers. More specifically, in the scientific context of metagenomic assembly of short DNA reads into "contigs," we examine the effects of degrading the quality of the training data by multiple mechanisms, and for four classifiers -- Bayes classifiers, neural nets, partition models and random forests. We investigate both individual behavior and congruence among the classifiers. We find breakdown-like behavior that holds for all four classifiers, as degradation increases and they move from being mostly correct to only coincidentally correct, because they are wrong in the same way. In the process, a picture of spatial heterogeneity emerges: as the training data move farther from analysis data, classifier decisions degenerate, the boundary becomes less dense, and congruence increases.