The Interaction of Entropy-Based Discretization and Sample Size: An Empirical Study

TL;DR

This empirical study investigates how sample size influences entropy-based discretization (CAIM) and its bias in data mining performance, revealing that smaller samples and certain attribute types can introduce optimistic bias.

Contribution

It provides the first empirical evidence that discretizing within cross-validation folds can bias performance metrics, especially with small sample sizes and high-dimensional data.

Findings

Discretization within cross-validation folds causes optimistic bias.

Smaller sample sizes increase bias in discretized models.

Attribute types and quantity influence the bias magnitude.

Abstract

An empirical investigation of the interaction of sample size and discretization - in this case the entropy-based method CAIM (Class-Attribute Interdependence Maximization) - was undertaken to evaluate the impact and potential bias introduced into data mining performance metrics due to variation in sample size as it impacts the discretization process. Of particular interest was the effect of discretizing within cross-validation folds averse to outside discretization folds. Previous publications have suggested that discretizing externally can bias performance results; however, a thorough review of the literature found no empirical evidence to support such an assertion. This investigation involved construction of over 117,000 models on seven distinct datasets from the UCI (University of California-Irvine) Machine Learning Library and multiple modeling methods across a variety of configurations of sample size and discretization, with each unique "setup" being independently replicated ten times. The analysis revealed a significant optimistic bias as sample sizes decreased and discretization was employed. The study also revealed that there may be a relationship between the interaction that produces such bias and the numbers and types of predictor attributes, extending the "curse of dimensionality" concept from feature selection into the discretization realm. Directions for further exploration are laid out, as well some general guidelines about the proper application of discretization in light of these results.