Layer-Stack Temperature Scaling

TL;DR

Layer-Stack Temperature Scaling (LATES) extends temperature scaling across all neural network layers, improving calibration and accuracy without retraining or extra data, and achieves state-of-the-art results when combined with Monte Carlo Dropout.

Contribution

This paper introduces LATES, a novel method that applies weighted voting across all layers during inference to enhance model calibration and accuracy.

Findings

LATES improves calibration and accuracy across multiple CNN architectures.

LATES combined with Monte Carlo Dropout matches state-of-the-art results on CIFAR datasets.

LATES requires no additional training or data, making it efficient and practical.

Abstract

Recent works demonstrate that early layers in a neural network contain useful information for prediction. Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy. We call this procedure "layer-stack temperature scaling" (LATES). Informally, LATES grants each layer a weighted vote during inference. We evaluate it on five popular convolutional neural network architectures both in- and out-of-distribution and observe a consistent improvement over temperature scaling in terms of accuracy, calibration, and AUC. All conclusions are supported by comprehensive statistical analyses. Since LATES neither retrains the architecture nor introduces many more parameters, its advantages can be reaped without requiring additional data beyond what is used in temperature scaling. Finally, we show that combining LATES with Monte Carlo Dropout matches state-of-the-art results on CIFAR10/100.