Random Forest Calibration

TL;DR

This paper systematically evaluates calibration methods for Random Forest classifiers, revealing that well-optimized RF models often outperform traditional calibration techniques, especially with limited data.

Contribution

It provides a comprehensive comparison of calibration methods for RF, demonstrating that optimized RF models can match or surpass calibration techniques.

Findings

Well-optimized RF models perform as well or better than calibration methods.

Traditional calibration methods offer limited improvements unless extensive data is available.

Systematic comparison reveals the impact of hyper-parameters on calibration quality.

Abstract

The Random Forest (RF) classifier is often claimed to be relatively well calibrated when compared with other machine learning methods. Moreover, the existing literature suggests that traditional calibration methods, such as isotonic regression, do not substantially enhance the calibration of RF probability estimates unless supplied with extensive calibration data sets, which can represent a significant obstacle in cases of limited data availability. Nevertheless, there seems to be no comprehensive study validating such claims and systematically comparing state-of-the-art calibration methods specifically for RF. To close this gap, we investigate a broad spectrum of calibration methods tailored to or at least applicable to RF, ranging from scaling techniques to more advanced algorithms. Our results based on synthetic as well as real-world data unravel the intricacies of RF probability estimates, scrutinize the impacts of hyper-parameters, compare calibration methods in a systematic way. We show that a well-optimized RF performs as well as or better than leading calibration approaches.