Quantile Regularization: Towards Implicit Calibration of Regression Models

TL;DR

This paper introduces a novel quantile regularizer for calibrating regression models during training, improving their predictive uncertainty estimates without needing extra datasets or post-processing.

Contribution

It proposes a trainable, end-to-end quantile regularizer based on KL divergence for better calibration of regression models, outperforming post-hoc methods.

Findings

Significantly improves calibration in regression models.

Works with various training objectives like Dropout VI and Deep Ensembles.

Outperforms post-hoc calibration methods in empirical tests.

Abstract

Recent works have shown that most deep learning models are often poorly calibrated, i.e., they may produce overconfident predictions that are wrong. It is therefore desirable to have models that produce predictive uncertainty estimates that are reliable. Several approaches have been proposed recently to calibrate classification models. However, there is relatively little work on calibrating regression models. We present a method for calibrating regression models based on a novel quantile regularizer defined as the cumulative KL divergence between two CDFs. Unlike most of the existing approaches for calibrating regression models, which are based on post-hoc processing of the model's output and require an additional dataset, our method is trainable in an end-to-end fashion without requiring an additional dataset. The proposed regularizer can be used with any training objective for regression. We also show that post-hoc calibration methods like Isotonic Calibration sometimes compound miscalibration whereas our method provides consistently better calibrations. We provide empirical results demonstrating that the proposed quantile regularizer significantly improves calibration for regression models trained using approaches, such as Dropout VI and Deep Ensembles.