Confidence Intervals for Evaluation of Data Mining

TL;DR

This paper develops fast, asymptotic confidence intervals for various data mining performance measures, enabling statistically sound comparisons of classification rules without extensive resampling.

Contribution

It introduces a novel 'blurring correction' for variance, extending the plus-four method to general performance measures in data mining.

Findings

Confidence intervals achieve good finite sample coverage.

The method allows simultaneous inference on multiple measures.

It avoids computationally intensive bootstrap resampling.

Abstract

In data mining, when binary prediction rules are used to predict a binary outcome, many performance measures are used in a vast array of literature for the purposes of evaluation and comparison. Some examples include classification accuracy, precision, recall, F measures, and Jaccard index. Typically, these performance measures are only approximately estimated from a finite dataset, which may lead to findings that are not statistically significant. In order to properly quantify such statistical uncertainty, it is important to provide confidence intervals associated with these estimated performance measures. We consider statistical inference about general performance measures used in data mining, with both individual and joint confidence intervals. These confidence intervals are based on asymptotic normal approximations and can be computed fast, without needs to do bootstrap resampling. We study the finite sample coverage probabilities for these confidence intervals and also propose a `blurring correction' on the variance to improve the finite sample performance. This 'blurring correction' generalizes the plus-four method from binomial proportion to general performance measures used in data mining. Our framework allows multiple performance measures of multiple classification rules to be inferred simultaneously for comparisons.