Calibrating a Deep Neural Network with Its Predecessors

TL;DR

This paper introduces Predecessor Combination Search (PCS), a novel regularization technique that improves neural network calibration and robustness by leveraging earlier training stages of individual network blocks.

Contribution

The paper proposes PCS, a new method that enhances calibration by combining block predecessors from different training stages, outperforming existing methods.

Findings

PCS achieves state-of-the-art calibration on multiple datasets.

PCS improves robustness under dataset distribution shifts.

Early stopping fails to calibrate networks despite mitigating overfitting.

Abstract

Confidence calibration - the process to calibrate the output probability distribution of neural networks - is essential for safety-critical applications of such networks. Recent works verify the link between mis-calibration and overfitting. However, early stopping, as a well-known technique to mitigate overfitting, fails to calibrate networks. In this work, we study the limitions of early stopping and comprehensively analyze the overfitting problem of a network considering each individual block. We then propose a novel regularization method, predecessor combination search (PCS), to improve calibration by searching a combination of best-fitting block predecessors, where block predecessors are the corresponding network blocks with weight parameters from earlier training stages. PCS achieves the state-of-the-art calibration performance on multiple datasets and architectures. In addition, PCS improves model robustness under dataset distribution shift.