Information-Theoretic Measures for Objective Evaluation of Classifications

TL;DR

This paper systematically studies objective, parameter-free information-theoretic measures for evaluating classification performance, proposing 24 normalized ITMs that are theoretically sound and applicable without cost input data.

Contribution

It introduces a set of 24 normalized, parameter-free information-theoretic measures for objective classification evaluation, with a framework for understanding their advantages and limitations.

Findings

ITMs are theoretically more sound than conventional measures.

ITMs can distinguish error and reject types without cost data.

The best ITM is identified and its properties are analytically derived.

Abstract

This work presents a systematic study of objective evaluations of abstaining classifications using Information-Theoretic Measures (ITMs). First, we define objective measures for which they do not depend on any free parameter. This definition provides technical simplicity for examining "objectivity" or "subjectivity" directly to classification evaluations. Second, we propose twenty four normalized ITMs, derived from either mutual information, divergence, or cross-entropy, for investigation. Contrary to conventional performance measures that apply empirical formulas based on users' intuitions or preferences, the ITMs are theoretically more sound for realizing objective evaluations of classifications. We apply them to distinguish "error types" and "reject types" in binary classifications without the need for input data of cost terms. Third, to better understand and select the ITMs, we suggest three desirable features for classification assessment measures, which appear more crucial and appealing from the viewpoint of classification applications. Using these features as "meta-measures", we can reveal the advantages and limitations of ITMs from a higher level of evaluation knowledge. Numerical examples are given to corroborate our claims and compare the differences among the proposed measures. The best measure is selected in terms of the meta-measures, and its specific properties regarding error types and reject types are analytically derived.