On the Calibration of Nested Dichotomies for Large Multiclass Tasks

TL;DR

This paper investigates the poor probability calibration of nested dichotomies in large multiclass tasks and demonstrates that calibrating both internal models and the overall structure significantly improves predictive accuracy and log-loss.

Contribution

It reveals calibration issues in nested dichotomies and proposes calibration strategies that enhance their performance in large multiclass classification.

Findings

Calibration of nested dichotomies is often poor, even with well-calibrated base models.

Calibrating internal models and the overall structure improves accuracy and log-loss.

Calibration strategies are especially effective when the number of classes is large.

Abstract

Nested dichotomies are used as a method of transforming a multiclass classification problem into a series of binary problems. A tree structure is induced that recursively splits the set of classes into subsets, and a binary classification model learns to discriminate between the two subsets of classes at each node. In this paper, we demonstrate that these nested dichotomies typically exhibit poor probability calibration, even when the base binary models are well calibrated. We also show that this problem is exacerbated when the binary models are poorly calibrated. We discuss the effectiveness of different calibration strategies and show that accuracy and log-loss can be significantly improved by calibrating both the internal base models and the full nested dichotomy structure, especially when the number of classes is high.