Classifier Ensemble for Efficient Uncertainty Calibration of Deep Neural Networks for Image Classification

TL;DR

This paper presents efficient metamodel-based classifier ensembles that significantly improve the calibration of deep neural networks in image classification, reducing errors without sacrificing accuracy.

Contribution

Introduces novel, simple ensemble techniques with metamodels that enhance calibration of deep neural networks more effectively than traditional methods.

Findings

Metamodel ensembles outperform traditional ensembles in calibration metrics.

Calibration errors are significantly reduced with minimal impact on accuracy.

Ensembles require fewer parameters and eliminate the need for separate calibration datasets.

Abstract

This paper investigates novel classifier ensemble techniques for uncertainty calibration applied to various deep neural networks for image classification. We evaluate both accuracy and calibration metrics, focusing on Expected Calibration Error (ECE) and Maximum Calibration Error (MCE). Our work compares different methods for building simple yet efficient classifier ensembles, including majority voting and several metamodel-based approaches. Our evaluation reveals that while state-of-the-art deep neural networks for image classification achieve high accuracy on standard datasets, they frequently suffer from significant calibration errors. Basic ensemble techniques like majority voting provide modest improvements, while metamodel-based ensembles consistently reduce ECE and MCE across all architectures. Notably, the largest of our compared metamodels demonstrate the most substantial calibration improvements, with minimal impact on accuracy. Moreover, classifier ensembles with metamodels outperform traditional model ensembles in calibration performance, while requiring significantly fewer parameters. In comparison to traditional post-hoc calibration methods, our approach removes the need for a separate calibration dataset. These findings underscore the potential of our proposed metamodel-based classifier ensembles as an efficient and effective approach to improving model calibration, thereby contributing to more reliable deep learning systems.