Robust Classification for Imprecise Environments

TL;DR

This paper introduces a robust hybrid classifier that maintains or surpasses the performance of the best classifiers across uncertain conditions, using a novel ROC convex hull method for analysis and comparison.

Contribution

It presents a new hybrid classification approach and a robust performance comparison method that adapt to imprecise target conditions and costs.

Findings

Hybrid classifier performs at least as well as the best classifier in any target condition.

The ROC convex hull method enables efficient, incremental performance analysis.

Empirical evidence supports the need for robust classifiers in real-world problems.

Abstract

In real-world environments it usually is difficult to specify target operating conditions precisely, for example, target misclassification costs. This uncertainty makes building robust classification systems problematic. We show that it is possible to build a hybrid classifier that will perform at least as well as the best available classifier for any target conditions. In some cases, the performance of the hybrid actually can surpass that of the best known classifier. This robust performance extends across a wide variety of comparison frameworks, including the optimization of metrics such as accuracy, expected cost, lift, precision, recall, and workforce utilization. The hybrid also is efficient to build, to store, and to update. The hybrid is based on a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull (ROCCH) method combines techniques from ROC analysis, decision analysis and computational geometry, and adapts them to the particulars of analyzing learned classifiers. The method is efficient and incremental, minimizes the management of classifier performance data, and allows for clear visual comparisons and sensitivity analyses. Finally, we point to empirical evidence that a robust hybrid classifier indeed is needed for many real-world problems.