Hierarchical confusion matrix for classification performance evaluation

TL;DR

This paper introduces a hierarchical confusion matrix for evaluating hierarchical classifiers, generalizing traditional measures to complex structures like DAGs and multi-path labels, and demonstrates its effectiveness on real-world data.

Contribution

It proposes a novel hierarchical confusion matrix concept, extending evaluation measures to all hierarchical classification types, including DAGs and multi-path labels.

Findings

Effective evaluation of hierarchical classifiers demonstrated

Comparison shows advantages over traditional measures

Implementation available on GitHub

Abstract

In this work we propose a novel concept of a hierarchical confusion matrix, opening the door for popular confusion matrix based (flat) evaluation measures from binary classification problems, while considering the peculiarities of hierarchical classification problems. We develop the concept to a generalized form and prove its applicability to all types of hierarchical classification problems including directed acyclic graphs, multi path labelling, and non mandatory leaf node prediction. Finally, we use measures based on the novel confusion matrix to evaluate models within a benchmark for three real world hierarchical classification applications and compare the results to established evaluation measures. The results outline the reasonability of this approach and its usefulness to evaluate hierarchical classification problems. The implementation of hierarchical confusion matrix is available on GitHub.